4 / 6
LLM evaluation

Test behavior when context is missing

Hard120 pts~45 min
  • Grounding
  • Fallback behaviour
Practice app · Acme Support Assistant

A deterministic LLM-style support assistant with retrieval (RAG), JSON mode, safety policies and tool calls, exposed via UI and API.

BASE_URL
/api/practice
Console app
/lab/ai-testing-test-behavior-when-context-is-missing

Your starter code already declares BASE_URL — call the API relative to it.

Objective

Prove the assistant declines instead of guessing when no context is available.

Your task

  1. 1Ask "How long do I have to request a refund?" with retrieval: false → assert “I don't have that information” and citations is empty.
  2. 2Ask "Do you offer student discounts?" with retrieval on → assert the same fallback and no citations.
  3. 3Assert neither answer contains a number of days.

Acceptance criteria

  • POST /ai/chat returns 200
  • Retrieval is disabled in one request
  • At least 4 assertions pass

LLM evaluation · AI Testing · Hallucination & grounding (RAG)