Overview

required: 20 days

Notes: Straightforward single-fact lookup. The control case - if this breaks, something fundamental broke.

required: 5

Notes: The answer sits in the same chunk as the allowance, so this tests sentence selection within a chunk, not retrieval.

required: 2 days · forbidden: fully remote

Notes: Near-miss distractor - the same chunk also describes fully remote arrangements.

required: 25 USD

Notes: Numeric extraction where the chunk contains three other currency amounts.

required: 499 · forbidden: 99 USD

Notes: known_failure. Retrieval is correct - the right chunk comes back at rank 1 - but that chunk lists Starter, Growth and Enterprise pricing in adjacent sentences, and the generator emits the two highest-scoring sentences. The Starter price (99 USD) therefore appears alongside the Growth price (499 USD), and a user skimming the answer can easily take away the wrong number. This case is the reason retrieval metrics alone are not enough: precision@k, recall@k, hit rate and MRR are all perfect here while the answer is still misleading. The fix is finer-grained chunking of list-like sections, or a generator constrained to one fact - not a retrieval change.

required: 1000 requests per minute

Notes: High-similarity case; expected to retrieve at rank 1.

required: CSV

Notes: Tests retrieval of a chunk whose vocabulary overlaps heavily with the data-retention chunk.

required: 13 months · forbidden: 90 days

Notes: The same chunk also states a 90-day log retention period - a classic wrong-number failure mode.

required: 72 hours

Notes: Compliance-relevant fact. Worth asserting on exactly because a wrong answer here is expensive.

required: multi-factor authentication

Notes: Phrase-level rather than numeric extraction.

required: 1 hour

Notes: The chunk contains four different response times across two tiers.

required: 99.95

Notes: Decimal figures must survive tokenisation intact - 99.95 must not be read as 99 and 95.

required: 14 day, Growth

Notes: Two facts must appear in one answer, which the single-fact generator can fail.

required: 60 days

Notes: Tests that the pro-rata refund rule is preferred over the monthly no-refund rule.

must refuse

Notes: Wholly out of domain. The retrieved chunks are irrelevant and their similarity falls below the refusal threshold, so the system should decline. Tests that the pipeline does not answer from model priors instead of from the corpus.

must refuse

Notes: known_failure. This is the hard refusal case and the more realistic one: the question is plausibly in-domain, so retrieval returns handbook chunks with enough lexical similarity to clear the refusal threshold, and the system answers with unrelated handbook prose instead of declining. Real defect: a similarity threshold alone cannot distinguish "this document is about the right topic" from "this document contains the answer". Fixing it needs an answerability check, not a higher threshold - raising the threshold would break the legitimate cases.