1 / 5
LLM evaluation

Identify sources of non-determinism

Easy30 pts~12 min
  • Non-determinism
  • Sampling
  • Invariants
Practice app · Acme Support Assistant

A deterministic LLM-style support assistant with retrieval (RAG), JSON mode, safety policies and tool calls, exposed via UI and API.

BASE_URL
/api/practice
Console app
/lab/ai-testing-identify-sources-of-non-determinism

Your starter code already declares BASE_URL — call the API relative to it.

Objective

Show which request settings change the assistant's wording and prove that the facts stay stable regardless.

Your task

  1. 1Ask "How long do I have to request a refund?" with temperature 0.9 and seeds 1, 2, 3, 4, 5.
  2. 2Assert every answer contains “30 days” and cites kb-refunds, even if the wording differs.
  3. 3Send seed 3 at temperature 0.9 twice and assert both output_text values are identical.
  4. 4Record which variables (temperature, seed, model, retrieval) affect the output.

Acceptance criteria

  • POST /ai/chat returns 200
  • Requests vary seed and temperature
  • At least 3 assertions pass

LLM evaluation · AI Testing · Foundations of AI testing