3 / 5
LLM evaluation

Define acceptance criteria for a generative feature

Medium70 pts~25 min
  • Acceptance criteria
  • Quality dimensions
Practice app · Acme Support Assistant

A deterministic LLM-style support assistant with retrieval (RAG), JSON mode, safety policies and tool calls, exposed via UI and API.

BASE_URL
/api/practice
Console app
/lab/ai-testing-define-acceptance-criteria-for-a-generative-feature

Your starter code already declares BASE_URL — call the API relative to it.

Objective

Turn a vague requirement (“answers shipping questions well”) into concrete, automatable acceptance checks.

Your task

  1. 1POST BASE_URL + "/ai/chat" with { "messages": [{ "role": "user", "content": <prompt> }], "temperature": 0 } using "How long does standard shipping take?".
  2. 2Assert output_text contains “3–5 business days” (fact).
  3. 3Assert citations include kb-shipping (grounding) and refused is false (policy).
  4. 4Assert finish_reason === "stop" and latency_ms < 2000 (quality of service).

Acceptance criteria

  • POST /ai/chat returns 200
  • At least 5 assertions pass

LLM evaluation · AI Testing · Foundations of AI testing