RAGProbe regression report

Category	Passed	Pass rate	Mean score
billing	2/2	100.0%	0.996
hr-policy	4/4	100.0%	0.943
product	2/3	66.7%	0.946
refusal	1/2	50.0%	0.859
security	3/3	100.0%	0.992
support	2/2	100.0%	1.000

passpto-annual-allowanceHow many days of paid time off do full-time employees get each year?0.80

Answer

Full-time employees accrue 20 days of paid time off per calendar year. [employee_handbook#paid-time-off] Employees may carry over a maximum of 5 unused PTO days into the following calendar year; any balance above 5 days is forfeited on 31 December.

Checks

	Check	Score	Detail
fail	exact_match	0.00	answer differs from expected_answer
fail	fuzzy_match	0.44	similarity 0.44 vs threshold 0.60
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.98	all 1 claim(s) grounded (score 0.98)
pass	llm_judge_faithfulness (advisory)	0.98	stub judge: deterministic n-gram grounding heuristic over 3 retrieved chunk(s)

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

employee_handbook#paid-time-offrank 1 · score 0.330expected

Full-time employees accrue 20 days of paid time off per calendar year. Accrual begins on the first day of employment at a rate of 1.67 days per month. Employees may carry over a maximum of 5 unused PTO days into the following calendar year; any balance above 5 days is forfeited on 31 December. PTO requests must be submitted at least 10 business days in advance through the HR self-service portal, except for sick leave, which may be reported the same day.

employee_handbook#remote-workrank 2 · score 0.170not in golden set

Northwind Robotics operates a hybrid model. Employees are expected on site at least 2 days per week, normally Tuesday and Thursday. Fully remote arrangements require written approval from both the direct manager and the People Operations team. Remote employees must be reachable during the core hours of 10:00 to 16:00 in their registered time zone.

employee_handbook#code-of-conductrank 3 · score 0.142not in golden set

Employees are expected to act with integrity and treat colleagues, customers and partners with respect. Harassment, discrimination and retaliation are prohibited. Concerns may be raised with a manager, with People Operations, or anonymously through the ethics hotline. All reports are investigated and the company prohibits retaliation against anyone who raises a concern in good faith.

Faithfulness

Heuristic grounding score 0.98 over 1 claim(s). LLM judge (stub) scored 0.98, disagreement 0.00.

passpto-carry-over-limitHow many unused PTO days can be carried over into the next calendar year?0.99

Answer

Full-time employees accrue 20 days of paid time off per calendar year. [employee_handbook#paid-time-off] Employees may carry over a maximum of 5 unused PTO days into the following calendar year; any balance above 5 days is forfeited on 31 December.

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.98	all 1 claim(s) grounded (score 0.98)
pass	llm_judge_faithfulness (advisory)	0.98	stub judge: deterministic n-gram grounding heuristic over 3 retrieved chunk(s)

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

employee_handbook#paid-time-offrank 1 · score 0.362expected

Full-time employees accrue 20 days of paid time off per calendar year. Accrual begins on the first day of employment at a rate of 1.67 days per month. Employees may carry over a maximum of 5 unused PTO days into the following calendar year; any balance above 5 days is forfeited on 31 December. PTO requests must be submitted at least 10 business days in advance through the HR self-service portal, except for sick leave, which may be reported the same day.

employee_handbook#equipment-and-hardwarerank 2 · score 0.192not in golden set

Every new employee receives a laptop, an external monitor, and a headset. The standard refresh cycle for laptops is 36 months. Lost or stolen equipment must be reported to the IT service desk within 24 hours so that the device can be remotely wiped. Personal use of company hardware is permitted within the limits of the acceptable use policy.

refund_policy#billing-disputesrank 3 · score 0.103not in golden set

Invoice disputes must be raised within 30 days of the invoice date. The billing team responds to disputes within 5 business days. Disputed amounts are not considered overdue while the dispute is open, and no late fees accrue on the disputed portion.

Faithfulness

Heuristic grounding score 0.98 over 1 claim(s). LLM judge (stub) scored 0.98, disagreement 0.00.

passremote-onsite-daysHow many days per week are employees expected to be on site?0.99

Answer

Employees are expected on site at least 2 days per week, normally Tuesday and Thursday. [employee_handbook#remote-work] Employees are expected to act with integrity and treat colleagues, customers and partners with respect. [employee_handbook#code-of-conduct]

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
pass	forbidden_absent	1.00	no forbidden keywords present
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 2 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.97	all 1 claim(s) grounded (score 0.97)
pass	llm_judge_faithfulness (advisory)	0.97	stub judge: deterministic n-gram grounding heuristic over 3 retrieved chunk(s)

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

employee_handbook#remote-workrank 1 · score 0.313expected

Northwind Robotics operates a hybrid model. Employees are expected on site at least 2 days per week, normally Tuesday and Thursday. Fully remote arrangements require written approval from both the direct manager and the People Operations team. Remote employees must be reachable during the core hours of 10:00 to 16:00 in their registered time zone.

employee_handbook#code-of-conductrank 2 · score 0.245not in golden set

Employees are expected to act with integrity and treat colleagues, customers and partners with respect. Harassment, discrimination and retaliation are prohibited. Concerns may be raised with a manager, with People Operations, or anonymously through the ethics hotline. All reports are investigated and the company prohibits retaliation against anyone who raises a concern in good faith.

product_faq#pricing-tiersrank 3 · score 0.168not in golden set

Northwind Fleet is sold in three tiers. The Starter tier costs 99 USD per month and supports up to 10 robots. The Growth tier costs 499 USD per month and supports up to 100 robots. The Enterprise tier is priced on request and has no robot limit. All tiers are billed monthly in arrears and include unlimited user seats.

Faithfulness

Heuristic grounding score 0.97 over 1 claim(s). LLM judge (stub) scored 0.97, disagreement 0.00.

passexpense-receipt-thresholdAbove what amount is a receipt required for an expense claim?0.99

Answer

Receipts are required for any single expense over 25 USD. [employee_handbook#expense-reimbursement] Expense reports are submitted through the expense portal and require approval from a cost centre owner.

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.96	all 1 claim(s) grounded (score 0.96)
pass	llm_judge_faithfulness (advisory)	0.96	stub judge: deterministic n-gram grounding heuristic over 3 retrieved chunk(s)

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	0.500

Retrieved chunks

employee_handbook#introrank 1 · score 0.200not in golden set

This handbook describes the employment policies of Northwind Robotics. It is a fictional document written for the RAGProbe sample corpus.

employee_handbook#expense-reimbursementrank 2 · score 0.181expected

Business expenses are reimbursed within 30 days of submission. Receipts are required for any single expense over 25 USD. Meals during business travel are capped at 75 USD per day. Airfare must be booked in economy class for flights under 6 hours. Expense reports are submitted through the expense portal and require approval from a cost centre owner.

employee_handbook#equipment-and-hardwarerank 3 · score 0.136not in golden set

Every new employee receives a laptop, an external monitor, and a headset. The standard refresh cycle for laptops is 36 months. Lost or stolen equipment must be reported to the IT service desk within 24 hours so that the device can be remotely wiped. Personal use of company hardware is permitted within the limits of the acceptable use policy.

Faithfulness

Heuristic grounding score 0.96 over 1 claim(s). LLM judge (stub) scored 0.96, disagreement 0.00.

failpricing-growth-tierHow much does the Growth tier cost per month?0.86

Answer

The Starter tier costs 99 USD per month and supports up to 10 robots. [product_faq#pricing-tiers] The Growth tier costs 499 USD per month and supports up to 100 robots.

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
fail	forbidden_absent	0.00	forbidden keyword(s) present: 99 USD
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	1.00	all 1 claim(s) grounded (score 1.00)
pass	llm_judge_faithfulness (advisory)	1.00	stub judge: deterministic n-gram grounding heuristic over 3 retrieved chunk(s)

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

product_faq#pricing-tiersrank 1 · score 0.195expected

Northwind Fleet is sold in three tiers. The Starter tier costs 99 USD per month and supports up to 10 robots. The Growth tier costs 499 USD per month and supports up to 100 robots. The Enterprise tier is priced on request and has no robot limit. All tiers are billed monthly in arrears and include unlimited user seats.

product_faq#api-rate-limitsrank 2 · score 0.161not in golden set

The Northwind Fleet REST API allows 1000 requests per minute on the Starter tier, 5000 requests per minute on the Growth tier, and a negotiated limit on Enterprise. Exceeding the limit returns HTTP 429 with a Retry-After header. Rate limits are applied per organisation, not per API key.

refund_policy#free-trialrank 3 · score 0.136not in golden set

New customers may start a 14 day free trial of the Growth tier. The trial does not require a credit card. At the end of the trial the account reverts to a read-only state unless a paid subscription is started. Trial accounts are limited to 5 robots and do not include the DataVault integration.

Faithfulness

Heuristic grounding score 1.00 over 1 claim(s). LLM judge (stub) scored 1.00, disagreement 0.00.

passapi-rate-limit-starterWhat is the API rate limit on the Starter tier?0.99

Answer

The Northwind Fleet REST API allows 1000 requests per minute on the Starter tier, 5000 requests per minute on the Growth tier, and a negotiated limit on Enterprise. [product_faq#api-rate-limits] Rate limits are applied per organisation, not per API key.

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.98	all 1 claim(s) grounded (score 0.98)
pass	llm_judge_faithfulness (advisory)	0.98	stub judge: deterministic n-gram grounding heuristic over 3 retrieved chunk(s)

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

product_faq#api-rate-limitsrank 1 · score 0.405expected

The Northwind Fleet REST API allows 1000 requests per minute on the Starter tier, 5000 requests per minute on the Growth tier, and a negotiated limit on Enterprise. Exceeding the limit returns HTTP 429 with a Retry-After header. Rate limits are applied per organisation, not per API key.

product_faq#pricing-tiersrank 2 · score 0.207not in golden set

Northwind Fleet is sold in three tiers. The Starter tier costs 99 USD per month and supports up to 10 robots. The Growth tier costs 499 USD per month and supports up to 100 robots. The Enterprise tier is priced on request and has no robot limit. All tiers are billed monthly in arrears and include unlimited user seats.

support_sla#uptime-commitmentrank 3 · score 0.161not in golden set

Northwind Fleet commits to 99.9 percent monthly uptime for the Growth tier and 99.95 percent for the Enterprise tier. Uptime is measured against the control plane API. Scheduled maintenance windows are excluded from the calculation and are announced at least 7 days in advance.

Faithfulness

Heuristic grounding score 0.98 over 1 claim(s). LLM judge (stub) scored 0.98, disagreement 0.00.

passdata-export-formatsWhich file formats can telemetry data be exported in?0.99

Answer

Telemetry data is retained for 13 months and then permanently deleted. [security_policy#data-retention] Customers can export their telemetry data at any time in CSV or Parquet format. [product_faq#data-export]

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 2 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.96	all 1 claim(s) grounded (score 0.96)
pass	llm_judge_faithfulness (advisory)	0.96	stub judge: deterministic n-gram grounding heuristic over 3 retrieved chunk(s)

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	0.500

Retrieved chunks

security_policy#data-retentionrank 1 · score 0.193not in golden set

Telemetry data is retained for 13 months and then permanently deleted. Application logs are retained for 90 days. Customer account records are retained for the duration of the contract plus 7 years to satisfy financial reporting obligations. Customers may request earlier deletion of telemetry data through a support ticket.

product_faq#data-exportrank 2 · score 0.166expected

Customers can export their telemetry data at any time in CSV or Parquet format. Exports are generated asynchronously and a download link is emailed when the archive is ready. Export archives remain available for 7 days. There is no charge for data export on any tier.

employee_handbook#paid-time-offrank 3 · score 0.153not in golden set

Full-time employees accrue 20 days of paid time off per calendar year. Accrual begins on the first day of employment at a rate of 1.67 days per month. Employees may carry over a maximum of 5 unused PTO days into the following calendar year; any balance above 5 days is forfeited on 31 December. PTO requests must be submitted at least 10 business days in advance through the HR self-service portal, except for sick leave, which may be reported the same day.

Faithfulness

Heuristic grounding score 0.96 over 1 claim(s). LLM judge (stub) scored 0.96, disagreement 0.00.

passtelemetry-retention-periodHow long is telemetry data retained before deletion?0.99

Answer

Telemetry data is retained for 13 months and then permanently deleted. [security_policy#data-retention] Customers may request earlier deletion of telemetry data through a support ticket.

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
pass	forbidden_absent	1.00	no forbidden keywords present
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.96	all 1 claim(s) grounded (score 0.96)
pass	llm_judge_faithfulness (advisory)	0.96	stub judge: deterministic n-gram grounding heuristic over 3 retrieved chunk(s)

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

security_policy#data-retentionrank 1 · score 0.274expected

Telemetry data is retained for 13 months and then permanently deleted. Application logs are retained for 90 days. Customer account records are retained for the duration of the contract plus 7 years to satisfy financial reporting obligations. Customers may request earlier deletion of telemetry data through a support ticket.

product_faq#what-is-northwind-fleetrank 2 · score 0.155not in golden set

Northwind Fleet is a fleet management platform for autonomous warehouse robots. It provides live telemetry, task scheduling, battery management and a simulation environment for testing routing changes before they reach production hardware.

employee_handbook#paid-time-offrank 3 · score 0.127not in golden set

Full-time employees accrue 20 days of paid time off per calendar year. Accrual begins on the first day of employment at a rate of 1.67 days per month. Employees may carry over a maximum of 5 unused PTO days into the following calendar year; any balance above 5 days is forfeited on 31 December. PTO requests must be submitted at least 10 business days in advance through the HR self-service portal, except for sick leave, which may be reported the same day.

Faithfulness

Heuristic grounding score 0.96 over 1 claim(s). LLM judge (stub) scored 0.96, disagreement 0.00.

passbreach-notification-windowHow quickly are customers notified after a confirmed data breach?0.99

Answer

Customers affected by a confirmed data breach are notified within 72 hours. [security_policy#incident-response] Exceeding the limit returns HTTP 429 with a Retry-After header. [product_faq#api-rate-limits]

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 2 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.96	all 1 claim(s) grounded (score 0.96)
pass	llm_judge_faithfulness (advisory)	0.96	stub judge: deterministic n-gram grounding heuristic over 3 retrieved chunk(s)

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

security_policy#incident-responserank 1 · score 0.312expected

Security incidents are triaged within 1 hour of detection. Customers affected by a confirmed data breach are notified within 72 hours. The incident response team maintains a runbook for each incident class and conducts a blameless post-mortem within 5 business days of resolution.

product_faq#api-rate-limitsrank 2 · score 0.221not in golden set

The Northwind Fleet REST API allows 1000 requests per minute on the Starter tier, 5000 requests per minute on the Growth tier, and a negotiated limit on Enterprise. Exceeding the limit returns HTTP 429 with a Retry-After header. Rate limits are applied per organisation, not per API key.

support_sla#escalation-pathrank 3 · score 0.193not in golden set

If a Severity 1 incident is not resolved within 4 hours, it is automatically escalated to the engineering on-call lead. After 8 hours it is escalated to the director of engineering. Customers may request a manual escalation at any time by replying to the ticket with the word ESCALATE.

Faithfulness

Heuristic grounding score 0.96 over 1 claim(s). LLM judge (stub) scored 0.96, disagreement 0.00.

passproduction-access-mfaWhat authentication is required for access to production systems?1.00

Answer

Access to production systems follows the principle of least privilege. [security_policy#access-control] All production access requires multi-factor authentication and is granted through short-lived credentials that expire after 8 hours.

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	1.00	all 1 claim(s) grounded (score 1.00)
pass	llm_judge_faithfulness (advisory)	1.00	stub judge: deterministic n-gram grounding heuristic over 3 retrieved chunk(s)

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

security_policy#access-controlrank 1 · score 0.340expected

Access to production systems follows the principle of least privilege. All production access requires multi-factor authentication and is granted through short-lived credentials that expire after 8 hours. Access reviews are performed quarterly by the security team, and any account inactive for 60 days is automatically disabled.

product_faq#what-is-northwind-fleetrank 2 · score 0.158not in golden set

Northwind Fleet is a fleet management platform for autonomous warehouse robots. It provides live telemetry, task scheduling, battery management and a simulation environment for testing routing changes before they reach production hardware.

employee_handbook#introrank 3 · score 0.097not in golden set

This handbook describes the employment policies of Northwind Robotics. It is a fictional document written for the RAGProbe sample corpus.

Faithfulness

Heuristic grounding score 1.00 over 1 claim(s). LLM judge (stub) scored 1.00, disagreement 0.00.

passsev1-premium-response-timeWhat is the first response time for a Severity 1 incident on Premium support?1.00

Answer

Severity 1 incidents receive a first response within 1 hour on Premium support and within 4 business hours on Standard support. [support_sla#response-times] Severity 2 receives a response within 4 hours on Premium and 1 business day on Standard.

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	1.00	all 1 claim(s) grounded (score 1.00)
pass	llm_judge_faithfulness (advisory)	1.00	stub judge: deterministic n-gram grounding heuristic over 3 retrieved chunk(s)

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

support_sla#response-timesrank 1 · score 0.498expected

Severity 1 incidents receive a first response within 1 hour on Premium support and within 4 business hours on Standard support. Severity 2 receives a response within 4 hours on Premium and 1 business day on Standard. Severity 3 and 4 receive a response within 2 business days on both tiers.

support_sla#escalation-pathrank 2 · score 0.238not in golden set

If a Severity 1 incident is not resolved within 4 hours, it is automatically escalated to the engineering on-call lead. After 8 hours it is escalated to the director of engineering. Customers may request a manual escalation at any time by replying to the ticket with the word ESCALATE.

security_policy#incident-responserank 3 · score 0.237not in golden set

Security incidents are triaged within 1 hour of detection. Customers affected by a confirmed data breach are notified within 72 hours. The incident response team maintains a runbook for each incident class and conducts a blameless post-mortem within 5 business days of resolution.

Faithfulness

Heuristic grounding score 1.00 over 1 claim(s). LLM judge (stub) scored 1.00, disagreement 0.00.

passenterprise-uptime-commitmentWhat uptime does the Enterprise tier commit to?1.00

Answer

Northwind Fleet commits to 99.9 percent monthly uptime for the Growth tier and 99.95 percent for the Enterprise tier. [support_sla#uptime-commitment] Uptime is measured against the control plane API.

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	1.00	all 1 claim(s) grounded (score 1.00)
pass	llm_judge_faithfulness (advisory)	1.00	stub judge: deterministic n-gram grounding heuristic over 3 retrieved chunk(s)

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

support_sla#uptime-commitmentrank 1 · score 0.199expected

Northwind Fleet commits to 99.9 percent monthly uptime for the Growth tier and 99.95 percent for the Enterprise tier. Uptime is measured against the control plane API. Scheduled maintenance windows are excluded from the calculation and are announced at least 7 days in advance.

refund_policy#free-trialrank 2 · score 0.140not in golden set

New customers may start a 14 day free trial of the Growth tier. The trial does not require a credit card. At the end of the trial the account reverts to a read-only state unless a paid subscription is started. Trial accounts are limited to 5 robots and do not include the DataVault integration.

employee_handbook#code-of-conductrank 3 · score 0.121not in golden set

Employees are expected to act with integrity and treat colleagues, customers and partners with respect. Harassment, discrimination and retaliation are prohibited. Concerns may be raised with a manager, with People Operations, or anonymously through the ethics hotline. All reports are investigated and the company prohibits retaliation against anyone who raises a concern in good faith.

Faithfulness

Heuristic grounding score 1.00 over 1 claim(s). LLM judge (stub) scored 1.00, disagreement 0.00.

passfree-trial-termsHow long is the free trial and which tier does it run on?1.00

Answer

New customers may start a 14 day free trial of the Growth tier. [refund_policy#free-trial] The trial does not require a credit card.

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 2 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	1.00	all 1 claim(s) grounded (score 1.00)
pass	llm_judge_faithfulness (advisory)	1.00	stub judge: deterministic n-gram grounding heuristic over 3 retrieved chunk(s)

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

refund_policy#free-trialrank 1 · score 0.343expected

New customers may start a 14 day free trial of the Growth tier. The trial does not require a credit card. At the end of the trial the account reverts to a read-only state unless a paid subscription is started. Trial accounts are limited to 5 robots and do not include the DataVault integration.

product_faq#api-rate-limitsrank 2 · score 0.163not in golden set

The Northwind Fleet REST API allows 1000 requests per minute on the Starter tier, 5000 requests per minute on the Growth tier, and a negotiated limit on Enterprise. Exceeding the limit returns HTTP 429 with a Retry-After header. Rate limits are applied per organisation, not per API key.

employee_handbook#introrank 3 · score 0.156not in golden set

This handbook describes the employment policies of Northwind Robotics. It is a fictional document written for the RAGProbe sample corpus.

Faithfulness

Heuristic grounding score 1.00 over 1 claim(s). LLM judge (stub) scored 1.00, disagreement 0.00.

passannual-refund-windowCan an annual prepaid subscription be refunded, and within what period?0.99

Answer

Annual prepaid subscriptions may be refunded on a pro-rata basis within the first 60 days of the contract term. [refund_policy#refund-eligibility] Lost or stolen equipment must be reported to the IT service desk within 24 hours so that the device can be remotely wiped. [employee_handbook#equipment-and-hardware]

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 2 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.98	all 1 claim(s) grounded (score 0.98)
pass	llm_judge_faithfulness (advisory)	0.98	stub judge: deterministic n-gram grounding heuristic over 3 retrieved chunk(s)

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

refund_policy#refund-eligibilityrank 1 · score 0.225expected

Monthly subscriptions are not refundable for partial months. Annual prepaid subscriptions may be refunded on a pro-rata basis within the first 60 days of the contract term. Refunds are issued to the original payment method within 14 business days of approval.

employee_handbook#equipment-and-hardwarerank 2 · score 0.178not in golden set

Every new employee receives a laptop, an external monitor, and a headset. The standard refresh cycle for laptops is 36 months. Lost or stolen equipment must be reported to the IT service desk within 24 hours so that the device can be remotely wiped. Personal use of company hardware is permitted within the limits of the acceptable use policy.

refund_policy#free-trialrank 3 · score 0.118not in golden set

New customers may start a 14 day free trial of the Growth tier. The trial does not require a credit card. At the end of the trial the account reverts to a read-only state unless a paid subscription is started. Trial accounts are limited to 5 robots and do not include the DataVault integration.

Faithfulness

Heuristic grounding score 0.98 over 1 claim(s). LLM judge (stub) scored 0.98, disagreement 0.00.

passrefusal-out-of-domainWhat is the capital of France?1.00

Answer

I do not have enough information in the provided documents to answer that.

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
skip	keyword_presence	0.00	no required_keywords in dataset
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly refused
skip	citation_present	0.00	refusals are not expected to cite sources
skip	expected_citation_overlap	0.00	no expected_chunks in dataset
skip	grounding	0.00	skipped: refusals are not grounded in retrieved text by design
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	n/a
precision@3	n/a
recall@3	n/a
reciprocal_rank	n/a

Retrieved chunks

product_faq#data-exportrank 1 · score 0.195not in golden set

Customers can export their telemetry data at any time in CSV or Parquet format. Exports are generated asynchronously and a download link is emailed when the archive is ready. Export archives remain available for 7 days. There is no charge for data export on any tier.

refund_policy#cancellationrank 2 · score 0.171not in golden set

Subscriptions may be cancelled at any time from the billing settings page. Cancellation takes effect at the end of the current billing period, and the account remains fully functional until that date. There is no cancellation fee on any tier.

employee_handbook#expense-reimbursementrank 3 · score 0.161not in golden set

Business expenses are reimbursed within 30 days of submission. Receipts are required for any single expense over 25 USD. Meals during business travel are capped at 75 USD per day. Airfare must be booked in economy class for flights under 6 hours. Expense reports are submitted through the expense portal and require approval from a cost centre owner.

Faithfulness

Heuristic grounding score 0.00 over 1 claim(s).

Ungrounded claim(s):

I do not have enough information in the provided documents to answer that.

failrefusal-in-domain-but-absentWhat is the chief executive of Northwind Robotics paid?0.72

Answer

This handbook describes the employment policies of Northwind Robotics. [employee_handbook#intro] Northwind Robotics operates a hybrid model. [employee_handbook#remote-work]

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
skip	keyword_presence	0.00	no required_keywords in dataset
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
fail	refusal_behaviour	0.00	should have refused but answered
pass	citation_present	1.00	cites 2 retrieved chunk(s)
skip	expected_citation_overlap	0.00	no expected_chunks in dataset
pass	grounding	0.94	all 1 claim(s) grounded (score 0.94)
pass	llm_judge_faithfulness (advisory)	0.94	stub judge: deterministic n-gram grounding heuristic over 3 retrieved chunk(s)

Retrieval metrics

Metric	Value
hit_rate@3	n/a
precision@3	n/a
recall@3	n/a
reciprocal_rank	n/a

Retrieved chunks

employee_handbook#introrank 1 · score 0.212not in golden set

This handbook describes the employment policies of Northwind Robotics. It is a fictional document written for the RAGProbe sample corpus.

refund_policy#free-trialrank 2 · score 0.094not in golden set

New customers may start a 14 day free trial of the Growth tier. The trial does not require a credit card. At the end of the trial the account reverts to a read-only state unless a paid subscription is started. Trial accounts are limited to 5 robots and do not include the DataVault integration.

employee_handbook#remote-workrank 3 · score 0.093not in golden set

Northwind Robotics operates a hybrid model. Employees are expected on site at least 2 days per week, normally Tuesday and Thursday. Fully remote arrangements require written approval from both the direct manager and the People Operations team. Remote employees must be reachable during the core hours of 10:00 to 16:00 in their registered time zone.

Faithfulness

Heuristic grounding score 0.94 over 1 claim(s). LLM judge (stub) scored 0.94, disagreement 0.00.

Measure	Baseline	Current	Delta
Pass rate	87.5%	87.5%	+0.000
Mean score	0.956	0.956	+0.000
Passed	14	14	+0.000
Failed	2	2	+0.000

flatannual-refund-windowCan an annual prepaid subscription be refunded, and within what period?+0.000

no material change

Baseline answer

Annual prepaid subscriptions may be refunded on a pro-rata basis within the first 60 days of the contract term. [refund_policy#refund-eligibility] Lost or stolen equipment must be reported to the IT service desk within 24 hours so that the device can be remotely wiped. [employee_handbook#equipment-and-hardware]

Current answer

Annual prepaid subscriptions may be refunded on a pro-rata basis within the first 60 days of the contract term. [refund_policy#refund-eligibility] Lost or stolen equipment must be reported to the IT service desk within 24 hours so that the device can be remotely wiped. [employee_handbook#equipment-and-hardware]

flatapi-rate-limit-starterWhat is the API rate limit on the Starter tier?+0.000

no material change

Baseline answer

The Northwind Fleet REST API allows 1000 requests per minute on the Starter tier, 5000 requests per minute on the Growth tier, and a negotiated limit on Enterprise. [product_faq#api-rate-limits] Rate limits are applied per organisation, not per API key.

Current answer

The Northwind Fleet REST API allows 1000 requests per minute on the Starter tier, 5000 requests per minute on the Growth tier, and a negotiated limit on Enterprise. [product_faq#api-rate-limits] Rate limits are applied per organisation, not per API key.

flatbreach-notification-windowHow quickly are customers notified after a confirmed data breach?+0.000

no material change

Baseline answer

Customers affected by a confirmed data breach are notified within 72 hours. [security_policy#incident-response] Exceeding the limit returns HTTP 429 with a Retry-After header. [product_faq#api-rate-limits]

Current answer

Customers affected by a confirmed data breach are notified within 72 hours. [security_policy#incident-response] Exceeding the limit returns HTTP 429 with a Retry-After header. [product_faq#api-rate-limits]

flatdata-export-formatsWhich file formats can telemetry data be exported in?+0.000

no material change

Baseline answer

Telemetry data is retained for 13 months and then permanently deleted. [security_policy#data-retention] Customers can export their telemetry data at any time in CSV or Parquet format. [product_faq#data-export]

Current answer

Telemetry data is retained for 13 months and then permanently deleted. [security_policy#data-retention] Customers can export their telemetry data at any time in CSV or Parquet format. [product_faq#data-export]

flatenterprise-uptime-commitmentWhat uptime does the Enterprise tier commit to?+0.000

no material change

Baseline answer

Northwind Fleet commits to 99.9 percent monthly uptime for the Growth tier and 99.95 percent for the Enterprise tier. [support_sla#uptime-commitment] Uptime is measured against the control plane API.

Current answer

Northwind Fleet commits to 99.9 percent monthly uptime for the Growth tier and 99.95 percent for the Enterprise tier. [support_sla#uptime-commitment] Uptime is measured against the control plane API.

flatexpense-receipt-thresholdAbove what amount is a receipt required for an expense claim?+0.000

no material change

Baseline answer

Receipts are required for any single expense over 25 USD. [employee_handbook#expense-reimbursement] Expense reports are submitted through the expense portal and require approval from a cost centre owner.

Current answer

Receipts are required for any single expense over 25 USD. [employee_handbook#expense-reimbursement] Expense reports are submitted through the expense portal and require approval from a cost centre owner.

flatfree-trial-termsHow long is the free trial and which tier does it run on?+0.000

no material change

Baseline answer

New customers may start a 14 day free trial of the Growth tier. [refund_policy#free-trial] The trial does not require a credit card.

Current answer

New customers may start a 14 day free trial of the Growth tier. [refund_policy#free-trial] The trial does not require a credit card.

flatpricing-growth-tierHow much does the Growth tier cost per month?+0.000

no material change

Baseline answer

The Starter tier costs 99 USD per month and supports up to 10 robots. [product_faq#pricing-tiers] The Growth tier costs 499 USD per month and supports up to 100 robots.

Current answer

The Starter tier costs 99 USD per month and supports up to 10 robots. [product_faq#pricing-tiers] The Growth tier costs 499 USD per month and supports up to 100 robots.

flatproduction-access-mfaWhat authentication is required for access to production systems?+0.000

no material change

Baseline answer

Access to production systems follows the principle of least privilege. [security_policy#access-control] All production access requires multi-factor authentication and is granted through short-lived credentials that expire after 8 hours.

Current answer

Access to production systems follows the principle of least privilege. [security_policy#access-control] All production access requires multi-factor authentication and is granted through short-lived credentials that expire after 8 hours.

flatpto-annual-allowanceHow many days of paid time off do full-time employees get each year?+0.000

no material change

Baseline answer

Full-time employees accrue 20 days of paid time off per calendar year. [employee_handbook#paid-time-off] Employees may carry over a maximum of 5 unused PTO days into the following calendar year; any balance above 5 days is forfeited on 31 December.

Current answer

Full-time employees accrue 20 days of paid time off per calendar year. [employee_handbook#paid-time-off] Employees may carry over a maximum of 5 unused PTO days into the following calendar year; any balance above 5 days is forfeited on 31 December.

flatpto-carry-over-limitHow many unused PTO days can be carried over into the next calendar year?+0.000

no material change

Baseline answer

Full-time employees accrue 20 days of paid time off per calendar year. [employee_handbook#paid-time-off] Employees may carry over a maximum of 5 unused PTO days into the following calendar year; any balance above 5 days is forfeited on 31 December.

Current answer

Full-time employees accrue 20 days of paid time off per calendar year. [employee_handbook#paid-time-off] Employees may carry over a maximum of 5 unused PTO days into the following calendar year; any balance above 5 days is forfeited on 31 December.

flatrefusal-in-domain-but-absentWhat is the chief executive of Northwind Robotics paid?+0.000

no material change

Baseline answer

This handbook describes the employment policies of Northwind Robotics. [employee_handbook#intro] Northwind Robotics operates a hybrid model. [employee_handbook#remote-work]

Current answer

This handbook describes the employment policies of Northwind Robotics. [employee_handbook#intro] Northwind Robotics operates a hybrid model. [employee_handbook#remote-work]

flatrefusal-out-of-domainWhat is the capital of France?+0.000

no material change

Baseline answer

I do not have enough information in the provided documents to answer that.

Current answer

I do not have enough information in the provided documents to answer that.

flatremote-onsite-daysHow many days per week are employees expected to be on site?+0.000

no material change

Baseline answer

Employees are expected on site at least 2 days per week, normally Tuesday and Thursday. [employee_handbook#remote-work] Employees are expected to act with integrity and treat colleagues, customers and partners with respect. [employee_handbook#code-of-conduct]

Current answer

Employees are expected on site at least 2 days per week, normally Tuesday and Thursday. [employee_handbook#remote-work] Employees are expected to act with integrity and treat colleagues, customers and partners with respect. [employee_handbook#code-of-conduct]

flatsev1-premium-response-timeWhat is the first response time for a Severity 1 incident on Premium support?+0.000

no material change

Baseline answer

Severity 1 incidents receive a first response within 1 hour on Premium support and within 4 business hours on Standard support. [support_sla#response-times] Severity 2 receives a response within 4 hours on Premium and 1 business day on Standard.

Current answer

Severity 1 incidents receive a first response within 1 hour on Premium support and within 4 business hours on Standard support. [support_sla#response-times] Severity 2 receives a response within 4 hours on Premium and 1 business day on Standard.

flattelemetry-retention-periodHow long is telemetry data retained before deletion?+0.000

no material change

Baseline answer

Telemetry data is retained for 13 months and then permanently deleted. [security_policy#data-retention] Customers may request earlier deletion of telemetry data through a support ticket.

Current answer

Telemetry data is retained for 13 months and then permanently deleted. [security_policy#data-retention] Customers may request earlier deletion of telemetry data through a support ticket.

RAGProbe regression report

Summary

Retrieval metrics

By category

Cases

Regression diff