Evaluation lifecycle

What is Evaluation in DataFact?

Evaluation is the process of reviewing how an Agent responds to representative questions, recording feedback, changing the relevant data or configuration, and running the questions again. DataFact currently provides visible feedback records, Q&A Management workflows, Test Sets, task status, and downloadable results. These features support a repeatable review process; they do not constitute an automatic accuracy benchmark.

A Test Set is a collection of plain-text questions run against an eligible Agent. It is useful for repeating a defined question set after a change to data, knowledge, model, instructions, or Agent configuration. The resulting task records show what ran and provide output for review.

The current Test Set interface does not define expected answers, score individual results, or compare runs automatically. A completed task therefore does not prove that its answers are correct or improved. Evaluation conclusions require a human reviewer or an external method with documented criteria, reference answers, and scoring rules.

This distinction matters because repeatability and correctness are different. Re-running the same questions makes change visible, while a valid quality judgment still depends on authoritative sources, the intended business definition, and criteria appropriate to the decision.

How DataFact uses Evaluation

flowchart TD
  A["Observe an answer"] --> B["Submit feedback"]
  B --> C["Review the question and context"]
  C --> D["Change data, knowledge, or Agent configuration"]
  D --> E["Run representative questions again"]
  E --> A

Feedback captures a user's judgment. Q&A Management supports review of returned records and visible handling state. Test Sets repeat plain-text questions against an eligible Agent and produce task records and downloadable results.

Example

A team changes a revenue definition in an Analysis Model, then runs the same finance questions again. Reviewers compare each answer with the approved definition and authoritative data; DataFact does not calculate an accuracy score for them.

Limitations and edge cases

• A completed run is not an accuracy result.
• Test questions alone do not define expected answers or scoring criteria.
• Feedback is a user's judgment and may require follow-up context.
• Different source snapshots can affect a comparison between runs.

Related concepts

• Knowledge and answer quality
• Datasets and Analysis Models
• Evidence and citations
• AI answer verification