Overview
RAGProbe - Playwright docs case study · 3 stored run(s)Latest
2026-09-19T16:51:49ZProvider openaiConfig 7b22c4525ae3Dataset 4aa7c778ad32Corpus 59 docs / 841 chunksLatest run
deltas compare with the previous stored runPass rate53.2%▼-6.5 pts vs prev
Mean score0.730▼-0.119 vs prev
Hit rate@30.707▬+0.000 vs prev
MRR0.592▬+0.000 vs prev
Precision@30.259▬+0.000 vs prev
Recall@30.655▬+0.000 vs prev
Cases run62▬+0 vs prev33 passed, 29 failed
Top findings
All insights- bad Weakest category: locators3/9 passing (33.3%). Failing: loc-recommended-locator, loc-strict-mode, loc-visible-opacity, loc-stable-definition (+2 more).
- bad Pass rate has fallen for 2 consecutive run(s)Now 53.2%, down from 59.7% on the previous run and 61.3% 2 runs ago.
- warn Failures split evenly between generation and retrieval14 a generation problem (retrieval found every expected chunk): loc-actionability-checks, loc-manual-assertion, to-global-timeout, par-retries-flag (+10 more); 14 a retrieval problem (no expected chunk was retrieved): loc-recommended-locator, loc-strict-mode, loc-visible-opacity, loc-stable-definition (+10 more); 1 mixed (retrieval was partial): cfg-default-reporter-ci.
Run log
3 run(s) in history, newest first| Run | Started (UTC) | Config | Provider | Pass rate | Mean score | Passed | Changed vs previous |
|---|---|---|---|---|---|---|---|
| #3 | 2026-09-19T16:51:49Z | 7b22c4525ae3 | openai nondeterministic | 53.2% | 0.730 | 33/62 | evaluation.judge_enabled: True -> False; generation.base_url: https://api.openai.com/v1 -> https://generativelanguage.googleapis.com/v1beta/openai ... |
| #2 | 2026-09-19T16:51:34Z | 3c48f2289860 | stub | 59.7% | 0.849 | 37/62 | retrieval.embedder: tfidf -> fastembed |
| #1 | 2026-09-19T16:51:10Z | b6440920c991 | stub | 61.3% | 0.845 | 38/62 | - |
Metrics over run history
3 run(s), oldest to newest; y-axis is zoomed to the datametric per runconfig fingerprint changeddataset fingerprint changedhover for the run timestamp and what changed
Pass rate
53.2%Mean score
0.730Hit rate@3
0.707MRR
0.592Precision@3
0.259Recall@3
0.655Where the latest run fails
categories sorted worst firstPass rate by category
bar colour: blue at 100%, yellow below, red under 50%
Check outcomes
failpassskippedsorted by failures; advisory checks never gate
Score distribution
62 cases, bins of 0.1Category detail
| Category | Passed | Pass rate | Prev | Mean score |
|---|---|---|---|---|
| locators | 3/9 | 33.3% | 33.3% | 0.627 |
| config | 6/14 | 42.9% | 71.4% | 0.683 |
| tooling | 3/6 | 50.0% | 66.7% | 0.645 |
| network | 5/10 | 50.0% | 80.0% | 0.771 |
| auth | 1/2 | 50.0% | 100.0% | 0.903 |
| parallel | 6/11 | 54.5% | 63.6% | 0.678 |
| timeouts | 5/6 | 83.3% | 50.0% | 0.870 |
| out-of-scope | 4/4 | 100.0% | 0.0% | 1.000 |
Cases in the latest run
regressed / improved chips compare with the previous run| Status▾ | Case▾ | Category▾ | Score▾ | Δ prev▾ | Failing checks▾ | Hit▾ | History▾ | ||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ▶fail degraded | loc-recommended-locatorWhich locator does Playwright recommend I use first when writing resilient tests? | locators | 0.495 | -0.171 | keyword_presenceexpected_citation_overlapgrounding | 0.0 | |||||||||||||||||||||||||||||||||||||||||
Question Which locator does Playwright recommend I use first when writing resilient tests? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: The guide says to prioritise user-facing attributes, page.getByRole() first. Actual answer Playwright prioritizes role, text, and test id locators when figuring out the best locator [codegen#introduction, best_practices#generate-locators, debug#picking-locators]. Attribution retrievalno expected chunk was retrieved (hit rate 0), so the generator never saw the evidence. Checks
Retrieval metrics hit_rate@3 0.000precision@3 0.000recall@3 0.000reciprocal_rank 0.000 Retrieved chunks codegen#introductionrank 1 · score 0.890not in golden set Playwright comes with the ability to generate tests for you as you perform actions in the browser and is a great way to quickly get started with testing. Playwright will look at your page and figure out the best locator, prioritizing [role, text and test id locators](./locators.md). If the generator finds multiple elements matching the locator, it will improve the locator to make it resilient that uniquely identify the target element. best_practices#generate-locatorsrank 2 · score 0.887not in golden set Playwright has a [test generator](./codegen.md) that can generate tests and pick locators for you. It will look at your page and figure out the best locator, prioritizing role, text and test id locators. If the generator finds multiple elements matching the locator, it will improve the locator to make it resilient and uniquely identify the target element, so you don't have to worry about failing tests due to locators. debug#picking-locatorsrank 3 · score 0.830not in golden set While debugging, you might need to choose a more resilient locator. You can do this by clicking on the **Pick Locator** button and hovering over any element in the browser window. While hovering over an element you will see the code needed to locate this element highlighted below. Clicking an element in the browser will add the locator into the field where you can then either tweak it or copy it into your code. <img height="1782" width="2784" alt="Picking locators" src="https://user-images.githubusercontent.com/13063165/212968640-ce82a027-9277-4bdf-b0a9-6282fb2becb7.png" /> Playwright will look at your page and figure out the best locator, prioritizing [role, text and test id locators](./locators.md). If Playwright finds multiple elements matching the locator, it will improve the locator to make it resilient and uniquely identify the target element, so you don't have to worry about failing tests due to locators. Expected but not retrieved: Score across history 1 flip(s), all across config changes #1 0.83#2 0.67#3 0.50 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass improved | loc-testid-tradeoffWhat is the downside of locating elements by test id? | locators | 0.928 | +0.098 | - | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question What is the downside of locating elements by test id? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: The chunk says test ids are the most resilient but not user facing. Actual answer The downside of locating elements by test id is that test ids are not user-facing [locators#locate-by-test-id]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 1.000 Retrieved chunks locators#locate-by-test-idrank 1 · score 0.821expected Testing by test ids is the most resilient way of testing as even if your text or role of the attribute changes, the test will still pass. QA's and developers should define explicit test ids and query them with page.getByTestId(). However testing by test ids is not user facing. If the role or text value is important to you then consider using user facing locators such as [role](#locate-by-role) and [text locators](#locate-by-text). For example, consider the following DOM structure. ```html <button data-testid="directions">Itinéraire</button> ``` You can locate the element by its test id: ```js await page.getByTestId('directions').click(); ``` You can also use test ids when you choose to use the test id methodology or when you can't locate by [role](#locate-by-role) or [text](#locate-by-text). locators#set-a-custom-test-id-attributerank 2 · score 0.792not in golden set By default, page.getByTestId() will locate elements based on the `data-testid` attribute, but you can configure it in your test config or by calling selectors.setTestIdAttribute(). Set the test id to use a custom data attribute for your tests. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ use: { testIdAttribute: 'data-pw' } }); ``` In your html you can now use `data-pw` as your test id instead of the default `data-testid`. ```html <button data-pw="directions">Itinéraire</button> ``` And then locate the element as you would normally do: ```js await page.getByTestId('directions').click(); ``` locators#get-by-test-idrank 3 · score 0.785not in golden set Use the page.getByTestId() method to locate an element in a list. You may need to modify the html and add a test id if you don't already have a test id. For example, consider the following DOM structure: ```html <ul> <li data-testid='apple'>apple</li> <li data-testid='banana'>banana</li> <li data-testid='orange'>orange</li> </ul> ``` Locate an item by its test id of "orange" and then click it. ```js await page.getByTestId('orange').click(); ``` Score across history 1 flip(s), all across config changes #1 0.66#2 0.83#3 0.93 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail degraded | loc-strict-modeMy click threw an error saying the locator resolved to more than one element. Why does Playwright do that? | locators | 0.000 | -0.661 | keyword_presencerefusal_behaviourexpected_citation_overlap | 0.0 | |||||||||||||||||||||||||||||||||||||||||
Question My click threw an error saying the locator resolved to more than one element. Why does Playwright do that? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: Strictness - operations throw when more than one element matches. Actual answer I do not have enough information in the provided documents to answer that. Attribution retrievalno expected chunk was retrieved (hit rate 0), so the generator never saw the evidence. Checks
Retrieval metrics hit_rate@3 0.000precision@3 0.000recall@3 0.000reciprocal_rank 0.000 Retrieved chunks locators#throws-an-error-if-more-than-onerank 1 · score 0.869not in golden set ```js await page.getByRole('button').click(); ``` On the other hand, Playwright understands when you perform a multiple-element operation, so the following call works perfectly fine when the locator resolves to multiple elements. debug#picking-locatorsrank 2 · score 0.836not in golden set While debugging, you might need to choose a more resilient locator. You can do this by clicking on the **Pick Locator** button and hovering over any element in the browser window. While hovering over an element you will see the code needed to locate this element highlighted below. Clicking an element in the browser will add the locator into the field where you can then either tweak it or copy it into your code. <img height="1782" width="2784" alt="Picking locators" src="https://user-images.githubusercontent.com/13063165/212968640-ce82a027-9277-4bdf-b0a9-6282fb2becb7.png" /> Playwright will look at your page and figure out the best locator, prioritizing [role, text and test id locators](./locators.md). If Playwright finds multiple elements matching the locator, it will improve the locator to make it resilient and uniquely identify the target element, so you don't have to worry about failing tests due to locators. locators#works-fine-with-multiple-elementsrank 3 · score 0.811not in golden set ```js await page.getByRole('button').count(); ``` You can explicitly opt-out from strictness check by telling Playwright which element to use when multiple elements match, through locator.first(), locator.last(), and locator.nth(). These methods are **not recommended** because when your page changes, Playwright may click on an element you did not intend. Instead, follow best practices above to create a locator that uniquely identifies the target element. Expected but not retrieved: Score across history no pass/fail flips #1 0.83#2 0.66#3 0.00 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass degraded | loc-css-vs-roleWhy shouldn't I select a button by its CSS class in my test? | locators | 0.925 | -0.075 | - | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question Why shouldn't I select a button by its CSS class in my test? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: Best practices - DOM structure changes, class names change, tests break. Actual answer Your DOM can easily change, so depending on your DOM structure (such as a CSS class) can lead to failing tests if a designer changes something [best_practices#prefer-user-facing-attributes-to-xpath-or-css-selectors]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 1.000 Retrieved chunks best_practices#prefer-user-facing-attributes-to-xpath-or-css-selectorsrank 1 · score 0.833expected Your DOM can easily change so having your tests depend on your DOM structure can lead to failing tests. For example consider selecting this button by its CSS classes. Should the designer change something then the class might change, thus breaking your test. ```js // 👎 page.locator('button.buttonIcon.episode-actions-later'); ``` Use locators that are resilient to changes in the DOM. ```js // 👍 page.getByRole('button', { name: 'submit' }); ``` other_locators#css-elements-matching-one-of-the-conditionsrank 2 · score 0.790not in golden set Comma-separated list of CSS selectors will match all elements that can be selected by one of the selectors in that list. ```js // Clicks a <button> that has either a "Log in" or "Sign in" text. await page.locator('button:has-text("Log in"), button:has-text("Sign in")').click(); ``` The `:is()` pseudo-class is a [CSS pseudo-class](https://developer.mozilla.org/en-US/docs/Web/CSS/:is) that may be useful for specifying a list of extra conditions on an element. other_locators#css-matching-only-visible-elementsrank 3 · score 0.755not in golden set Playwright supports the `:visible` pseudo class in CSS selectors. For example, `css=button` matches all the buttons on the page, while `css=button:visible` only matches visible buttons. This is useful to distinguish elements that are very similar but differ in visibility. Prefer the locator.visible() shortcut over the `:visible` pseudo-class. Consider a page with two buttons, first invisible and second visible. ```html <button style='display: none'>Invisible</button> <button>Visible</button> ``` * This will find both buttons and throw a [strictness](./locators.md#strictness) violation error: ```js await page.locator('button').click(); ``` ```java page.locator("button").click(); ``` ```python async await page.locator("button").click() ``` ```python sync page.locator("button").click() ``` ```csharp await page.Locator("button").ClickAsync(); ``` * This will only find a second button, because it is visible, and then click it. ```js await page.locator('button:visible').click(); ``` ```java page.locator("button:visible").click(); ``` ```python async await page.locator("button:visible").click() ``` ```python sync page.locator("button:visible").click() ``` ```csharp await page.Locator("button:visible").ClickAsync(); ``` Score across history no pass/fail flips #1 1.00#2 1.00#3 0.93 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail degraded | loc-visible-opacityDoes Playwright treat an element with opacity 0 as visible? | locators | 0.048 | -0.483 | exact_matchfuzzy_matchkeyword_presencerefusal_behaviourexpected_citation_overlap | 0.0 | |||||||||||||||||||||||||||||||||||||||||
Question Does Playwright treat an element with opacity 0 as visible? Expected Yes. Elements with opacity:0 are considered visible; zero-size and display:none elements are not. required: Notes: Counter-intuitive rule that people get wrong; the answer must contain the opacity clause. Actual answer I do not have enough information in the provided documents to answer that. Attribution retrievalno expected chunk was retrieved (hit rate 0), so the generator never saw the evidence. Checks
Retrieval metrics hit_rate@3 0.000precision@3 0.000recall@3 0.000reciprocal_rank 0.000 Retrieved chunks other_locators#css-locatorrank 1 · score 0.815not in golden set We recommend prioritizing [user-visible locators](./locators.md#quick-guide) like text or accessible role instead of using CSS that is tied to the implementation and could break when the page changes. Playwright can locate an element by CSS selector. ```js await page.locator('css=button').click(); ``` Playwright augments standard CSS selectors in two ways: * CSS selectors pierce open shadow DOM. * Playwright adds custom pseudo-classes like `:visible`, `:has-text()`, `:has()`, `:is()`, `:nth-match()` and more. other_locators#css-matching-only-visible-elementsrank 2 · score 0.791not in golden set Playwright supports the `:visible` pseudo class in CSS selectors. For example, `css=button` matches all the buttons on the page, while `css=button:visible` only matches visible buttons. This is useful to distinguish elements that are very similar but differ in visibility. Prefer the locator.visible() shortcut over the `:visible` pseudo-class. Consider a page with two buttons, first invisible and second visible. ```html <button style='display: none'>Invisible</button> <button>Visible</button> ``` * This will find both buttons and throw a [strictness](./locators.md#strictness) violation error: ```js await page.locator('button').click(); ``` ```java page.locator("button").click(); ``` ```python async await page.locator("button").click() ``` ```python sync page.locator("button").click() ``` ```csharp await page.Locator("button").ClickAsync(); ``` * This will only find a second button, because it is visible, and then click it. ```js await page.locator('button:visible').click(); ``` ```java page.locator("button:visible").click(); ``` ```python async await page.locator("button:visible").click() ``` ```python sync page.locator("button:visible").click() ``` ```csharp await page.Locator("button:visible").ClickAsync(); ``` debug#playwright-inspect-selectorrank 3 · score 0.781not in golden set Reveal element in the Elements panel. ```bash playwright.inspect('text=Log in') ``` Expected but not retrieved: Score across history 1 flip(s), all across config changes #1 0.78#2 0.53#3 0.05 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail degraded | loc-stable-definitionWhat does Playwright mean when it waits for an element to be "stable"? | locators | 0.519 | -0.145 | keyword_presenceexpected_citation_overlap | 0.0 | |||||||||||||||||||||||||||||||||||||||||
Question What does Playwright mean when it waits for an element to be "stable"? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: Same bounding box for at least two consecutive animation frames. Actual answer Playwright considers an element to be stable when it is not animating or has completed its animation [actionability#introduction]. Attribution retrievalno expected chunk was retrieved (hit rate 0), so the generator never saw the evidence. Checks
Retrieval metrics hit_rate@3 0.000precision@3 0.000recall@3 0.000reciprocal_rank 0.000 Retrieved chunks writing_tests#assertionsrank 1 · score 0.787not in golden set Playwright includes [test assertions](./test-assertions.md) in the form of `expect` function. To make an assertion, call `expect(value)` and choose a matcher that reflects the expectation. Playwright includes async matchers that wait until the expected condition is met. Using these matchers makes tests non-flaky and resilient. For example, this code waits until the page gets the title containing "Playwright": ```js await expect(page).toHaveTitle(/Playwright/); ``` Here are the most popular async assertions. For the complete list, see [assertions guide](./test-assertions.md): | Assertion | Description | | :- | :- | | locatorAssertions.toBeChecked() | Checkbox is checked | | locatorAssertions.toBeEnabled() | Control is enabled | | locatorAssertions.toBeVisible() | Element is visible | | locatorAssertions.toContainText() | Element contains text | | locatorAssertions.toHaveAttribute() | Element has attribute | | locatorAssertions.toHaveCount() | List of elements has given length | | locatorAssertions.toHaveText() | Element matches text | | locatorAssertions.toHaveValue() | Input element has value | | pageAssertions.toHaveTitle() | Page has title | | pageAssertions.toHaveURL() | Page has URL | Playwright also includes generic matchers like `toEqual`, `toContain`, `toBeTruthy` that can be used to assert any conditions. These assertions do not actionability#introductionrank 2 · score 0.782not in golden set Playwright performs a range of actionability checks on the elements before making actions to ensure these actions behave as expected. It auto-waits for all the relevant checks to pass and only then performs the requested action. If the required checks do not pass within the given `timeout`, action fails with the `TimeoutError`. For example, for locator.click(), Playwright will ensure that: - locator resolves to exactly one element - element is Visible - element is Stable, as in not animating or completed animation - element [Receives Events], as in not obscured by other elements - element is Enabled Here is the complete list of actionability checks performed for each action: | Action | Visible | Stable | [Receives Events] | Enabled | Editable | | :- | :-: | :-: | :-: | :-: | :-: | | locator.check() | Yes | Yes | Yes | Yes | - | | locator.click() | Yes | Yes | Yes | Yes | - | | locator.dblclick() | Yes | Yes | Yes | Yes | - | | locator.setChecked() | Yes | actionability#assertionsrank 3 · score 0.779not in golden set Playwright includes auto-retrying assertions that remove flakiness by waiting until the condition is met, similarly to auto-waiting before actions. | Assertion | Description | | :- | :- | | locatorAssertions.toBeAttached() | Element is attached | | locatorAssertions.toBeChecked() | Checkbox is checked | | locatorAssertions.toBeDisabled() | Element is disabled | | locatorAssertions.toBeEditable() | Element is editable | | locatorAssertions.toBeEmpty() | Container is empty | | locatorAssertions.toBeEnabled() | Element is enabled | | locatorAssertions.toBeFocused() | Element is focused | | locatorAssertions.toBeHidden() | Element is not visible | | locatorAssertions.toBeInViewport() | Element intersects viewport | | locatorAssertions.toBeVisible() | Element is visible | | locatorAssertions.toContainText() | Element contains text | | locatorAssertions.toHaveAttribute() | Element has a DOM attribute | | locatorAssertions.toHaveClass() | Element has a class property | | locatorAssertions.toHaveCount() | List has exact number of children | | locatorAssertions.toHaveCSS() | Element has CSS property | | locatorAssertions.toHaveId() | Element has an ID | | locatorAssertions.toHaveJSProperty() | Element has a JavaScript property | | locatorAssertions.toHaveText() | Element matches text | | locatorAssertions.toHaveValue() | Input has a value | | locatorAssertions.toHaveValues() | Select has Expected but not retrieved: Score across history 1 flip(s), all across config changes #1 1.00#2 0.66#3 0.52 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail degraded | loc-actionability-checksWhat checks does Playwright do automatically before clicking an element? | locators | 0.918 | -0.026 | grounding | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question What checks does Playwright do automatically before clicking an element? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: The auto-wait list - visible, stable, receives events, enabled. Actual answer Before clicking an element, Playwright ensures that the locator resolves to exactly one element, and that the element is visible, stable, receives events, and is enabled [actionability#introduction]. Additionally, locators perform auto-waiting to check that elements are visible and enabled before executing a click [best_practices#use-locators]. Attribution generationevery expected chunk was retrieved (recall 1.0) yet the case fails on grounding. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 0.333 Retrieved chunks best_practices#use-locatorsrank 1 · score 0.842not in golden set In order to write end to end tests we need to first find elements on the webpage. We can do this by using Playwright's built in [locators](./locators.md). Locators come with auto waiting and retry-ability. Auto waiting means that Playwright performs a range of actionability checks on the elements, such as ensuring the element is visible and enabled before it performs the click. To make tests resilient, we recommend prioritizing user-facing attributes and explicit contracts. ```js // 👍 page.getByRole('button', { name: 'submit' }); ``` debug#stepping-through-your-testsrank 2 · score 0.828not in golden set You can play, pause or step through each action of your test using the toolbar at the top of the Inspector. You can see the current action highlighted in the test code, and matching elements highlighted in the browser window. <img height="1736" width="2680" alt="Playwright Inspector and browser" src="https://user-images.githubusercontent.com/13063165/212936618-84b87acc-bc2e-46ed-994b-32b2ef742e60.png" /> actionability#introductionrank 3 · score 0.828expected Playwright performs a range of actionability checks on the elements before making actions to ensure these actions behave as expected. It auto-waits for all the relevant checks to pass and only then performs the requested action. If the required checks do not pass within the given `timeout`, action fails with the `TimeoutError`. For example, for locator.click(), Playwright will ensure that: - locator resolves to exactly one element - element is Visible - element is Stable, as in not animating or completed animation - element [Receives Events], as in not obscured by other elements - element is Enabled Here is the complete list of actionability checks performed for each action: | Action | Visible | Stable | [Receives Events] | Enabled | Editable | | :- | :-: | :-: | :-: | :-: | :-: | | locator.check() | Yes | Yes | Yes | Yes | - | | locator.click() | Yes | Yes | Yes | Yes | - | | locator.dblclick() | Yes | Yes | Yes | Yes | - | | locator.setChecked() | Yes | Score across history no pass/fail flips #1 0.77#2 0.94#3 0.92 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass degraded | loc-force-clickWhat does the force option on click actually skip? | locators | 0.925 | -0.071 | - | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question What does the force option on click actually skip? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: Disables non-essential actionability checks such as the receives-events check. Actual answer The `force` option disables non-essential actionability checks, such as verifying that the target element actually receives click events [actionability#forcing-actions, input#forcing-the-click]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 1.000 Retrieved chunks actionability#forcing-actionsrank 1 · score 0.792expected Some actions like locator.click() support `force` option that disables non-essential actionability checks, for example passing truthy `force` to locator.click() method will not check that the target element actually receives click events. input#forcing-the-clickrank 2 · score 0.772not in golden set Sometimes, apps use non-trivial logic where hovering the element overlays it with another element that intercepts the click. This behavior is indistinguishable from a bug where element gets covered and the click is dispatched elsewhere. If you know this is taking place, you can bypass the [actionability](./actionability.md) checks and force the click: ```js await page.getByRole('button').click({ force: true }); ``` input#mouse-clickrank 3 · score 0.733not in golden set Performs a simple human click. ```js // Generic click await page.getByRole('button').click(); // Double click await page.getByText('Item').dblclick(); // Right click await page.getByText('Item').click({ button: 'right' }); // Shift + click await page.getByText('Item').click({ modifiers: ['Shift'] }); // Ctrl + click on Windows and Linux // Meta + click on macOS await page.getByText('Item').click({ modifiers: ['ControlOrMeta'] }); // Hover over element await page.getByText('Item').hover(); // Click the top left corner await page.getByText('Item').click({ position: { x: 0, y: 0 } }); ``` Under the hood, this and other pointer-related methods: - wait for element with given selector to be in DOM - wait for it to become displayed, i.e. not empty, no `display:none`, no `visibility:hidden` - wait for it to stop moving, for example, until css transition finishes - scroll the element into view - wait for it to receive pointer events at the action point, for example, waits until element becomes non-obscured by other elements - retry if the element is detached during any of the above checks Score across history no pass/fail flips #1 1.00#2 1.00#3 0.92 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail regressed | loc-manual-assertionIs expect(await locator.isVisible()).toBe(true) a good assertion? | locators | 0.888 | -0.110 | grounding | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question Is expect(await locator.isVisible()).toBe(true) a good assertion? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: Should steer to web-first assertions; isVisible() does not wait. Actual answer No, that is a manual assertion and is not recommended because the test will not wait for the element and will return immediately [best_practices#don-t-use-manual-assertions]. Instead, you should use web-first assertions like `await expect(locator).toBeVisible()` [best_practices#don-t-use-manual-assertions]. Attribution generationevery expected chunk was retrieved (recall 1.0) yet the case fails on grounding. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 1.000 Retrieved chunks best_practices#don-t-use-manual-assertionsrank 1 · score 0.895expected Don't use manual assertions that are not awaiting the expect. In the code below the await is inside the expect rather than before it. When using assertions such as `isVisible()` the test won't wait a single second, it will just check the locator is there and return immediately. ```js // 👎 expect(await page.getByText('welcome').isVisible()).toBe(true); ``` Use web first assertions such as `toBeVisible()` instead. ```js // 👍 await expect(page.getByText('welcome')).toBeVisible(); ``` test_assertions#add-custom-matchers-using-expect-extend~2rank 2 · score 0.818not in golden set ${locator}\n` + `Expected: not ${this.utils.printExpected(expected)}\n` + (matcherResult ? `Received: ${this.utils.printReceived(matcherResult.actual)}` : '') : () => this.utils.matcherHint(assertionName, undefined, undefined, { isNot: this.isNot }) + '\n\n' + `Locator: ${locator}\n` + `Expected: ${this.utils.printExpected(expected)}\n` + (matcherResult ? `Received: ${this.utils.printReceived(matcherResult.actual)}` : ''); return { message, pass, name: assertionName, expected, actual: matcherResult?.actual, }; }, }); ``` Now we can use `toHaveAmount` in the test. ```js import { test, expect } from './fixtures'; test('amount', async () => { await expect(page.locator('.cart')).toHaveAmount(4); }); ``` test_assertions#custom-expect-messagerank 3 · score 0.813not in golden set You can specify a custom expect message as a second argument to the `expect` function, for example: ```js await expect(page.getByText('Name'), 'should be logged in').toBeVisible(); ``` This message will be shown in reporters, both for passing and failing expects, providing more context about the assertion. When expect passes, you might see a successful step like this: ```txt ✅ should be logged in @example.spec.ts:18 ``` When expect fails, the error would look like this: ```bash Error: should be logged in Call log: - expect.toBeVisible with timeout 5000ms - waiting for "getByText('Name')" 2 | 3 | test('example test', async({ page }) => { > 4 | await expect(page.getByText('Name'), 'should be logged in').toBeVisible(); | ^ 5 | }); 6 | ``` Soft assertions also support custom message: ```js expect.soft(value, 'my soft assertion').toBe(56); ``` Score across history 1 flip(s), all across config changes #1 1.00#2 1.00#3 0.89 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass improved | to-default-test-timeoutWhat is the default timeout for a single test? | timeouts | 0.673 | +0.165 | exact_matchfuzzy_match | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question What is the default timeout for a single test? Expected Each test has a 30 second timeout by default; fixture setup and beforeEach time count against it. required: Actual answer The default timeout for each test is 30 seconds [test_timeouts#test-timeout]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 0.500reciprocal_rank 0.333 Retrieved chunks test_timeouts#advanced-low-level-timeoutsrank 1 · score 0.810not in golden set These are the low-level timeouts that are pre-configured by the test runner, you should not need to change these. If you happen to be in this section because your test are flaky, it is very likely that you should be looking for the solution elsewhere. |Timeout |Default |Description | |:----------|:----------------|:--------------------------------| |Action timeout| no timeout |Timeout for each action<br/><span style={{textTransform:'uppercase',fontSize:'smaller',fontWeight:'bold',opacity:'0.7'}}>Set in config</span><br/><code>{`{ use: { actionTimeout: 10_000 } }`}</code><br/><span style={{textTransform: 'uppercase',fontSize: 'smaller', fontWeight: 'bold', opacity: '0.7'}}>Override in test</span><br/>`locator.click({ timeout: 10_000 })` | |Navigation timeout| no timeout |Timeout for each navigation action<br/><span style={{textTransform:'uppercase',fontSize:'smaller',fontWeight:'bold',opacity:'0.7'}}>Set in config</span><br/><code>{`{ use: { navigationTimeout: 30_000 } }`}</code><br/><span style={{textTransform: 'uppercase',fontSize: 'smaller', fontWeight: 'bold', opacity: '0.7'}}>Override in test</span><br/>`page.goto('/', { timeout: 30_000 })` | |Global timeout|no timeout |Global timeout for the whole test run<br/><span style={{textTransform:'uppercase',fontSize:'smaller',fontWeight:'bold',opacity:'0.7'}}>Set in config</span><br/>`{ globalTimeout: 3_600_000 }`<br/> | |`beforeAll`/`afterAll` timeout|30_000 ms|Timeout for the hook<br/><span style={{textTransform:'uppercase',fontSize:'smaller',fontWeight:'bold',opacity:'0.7'}}>Set in hook</span><br/>`test.setTimeout(60_000)`<br/> | |Fixture timeout|no timeout |Timeout for an individual fixture<br/><span style={{textTransform:'uppercase',fontSize:'smaller',fontWeight:'bold',opacity:'0.7'}}>Set in fixture</span><br/>`{ scope: 'test', timeout: 30_000 }`<br/> | test_timeouts#global-timeoutrank 2 · score 0.779not in golden set Playwright Test supports a timeout for the whole test run. This prevents excess resource usage when everything went wrong. There is no default global timeout, but you can set a reasonable one in the config, for example one hour. Global timeout produces the following error: ```txt Running 1000 tests using 10 workers 514 skipped 486 passed Timed out waiting 3600s for the entire test run ``` You can set global timeout in the config. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ globalTimeout: 3_600_000, }); ``` API reference: testConfig.globalTimeout. test_timeouts#test-timeoutrank 3 · score 0.763expected Playwright Test enforces a timeout for each test, 30 seconds by default. Time spent by the test function, fixture setups, and `beforeEach` hooks is included in the test timeout. Timed out test produces the following error: ```txt example.spec.ts:3:1 › basic test =========================== Timeout of 30000ms exceeded. ``` Additional separate timeout, of the same value, is shared between fixture teardowns and `afterEach` hooks, after the test function has finished. The same timeout value also applies to `beforeAll` and `afterAll` hooks, but they do not share time with any test. Expected but not retrieved: Score across history 1 flip(s), all across config changes #1 0.53#2 0.51#3 0.67 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass improved | to-expect-timeoutHow long does expect(locator).toHaveText() keep retrying before it fails? | timeouts | 0.873 | +0.088 | - | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question How long does expect(locator).toHaveText() keep retrying before it fails? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: 5 seconds, separate from the test timeout. The forbidden keyword catches an answer that conflates the two. Actual answer `locatorAssertions.toHaveText()` has a default timeout of 5 seconds [test_timeouts#expect-timeout]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 0.500reciprocal_rank 1.000 Retrieved chunks test_timeouts#expect-timeoutrank 1 · score 0.873expected Auto-retrying assertions like locatorAssertions.toHaveText() have a separate timeout, 5 seconds by default. Assertion timeout is unrelated to the test timeout. It produces the following error: ```txt example.spec.ts:3:1 › basic test =========================== Error: expect(received).toHaveText(expected) Expected string: "my text" Received string: "" Call log: - expect.toHaveText with timeout 5000ms - waiting for "locator('button')" ``` test_assertions#auto-retrying-assertionsrank 2 · score 0.810not in golden set The following assertions will retry until the assertion passes, or the assertion timeout is reached. Note that retrying assertions are async, so you must `await` them. | Assertion | Description | | :- | :- | | [await expect(locator).toBeAttached()](./api/class-locatorassertions.md#locator-assertions-to-be-attached) | Element is attached | | [await expect(locator).toBeChecked()](./api/class-locatorassertions.md#locator-assertions-to-be-checked) | Checkbox is checked | | [await expect(locator).toBeDisabled()](./api/class-locatorassertions.md#locator-assertions-to-be-disabled) | Element is disabled | | [await expect(locator).toBeEditable()](./api/class-locatorassertions.md#locator-assertions-to-be-editable) | Element is editable | | [await expect(locator).toBeEmpty()](./api/class-locatorassertions.md#locator-assertions-to-be-empty) | Container is empty | | [await expect(locator).toBeEnabled()](./api/class-locatorassertions.md#locator-assertions-to-be-enabled) | Element is enabled | | [await expect(locator).toBeFocused()](./api/class-locatorassertions.md#locator-assertions-to-be-focused) | Element is focused | | [await expect(locator).toBeHidden()](./api/class-locatorassertions.md#locator-assertions-to-be-hidden) | Element is not visible | | [await expect(locator).toBeInViewport()](./api/class-locatorassertions.md#locator-assertions-to-be-in-viewport) | Element intersects viewport | | [await expect(locator).toBeVisible()](./api/class-locatorassertions.md#locator-assertions-to-be-visible) | Element is visible | | [await expect(locator).toContainText()](./api/class-locatorassertions.md#locator-assertions-to-contain-text) | Element contains text | | [await expect(locator).toContainClass()](./api/class-locatorassertions.md#locator-assertions-to-contain-class) | Element has specified CSS classes | | [await expect(locator).toHaveAccessibleDescription()](./api/class-locatorassertions.md#locator-assertions-to-have-accessible-description) | Element has a matching [accessible description](https://w3c.github.io/accname/#dfn-accessible-description) | | [await expect(locator).toHaveAccessibleName()](./api/class-locatorassertions.md#locator-assertions-to-have-accessible-name) | Element has a matching [accessible name](https://w3c.github.io/accname/#dfn-accessible-name) | | [await expect(locator).toHaveAttribute()](./api/class-locatorassertions.md#locator-assertions-to-have-attribute) | Element has a DOM attribute | | [await expect(locator).toHaveClass()](./api/class-locatorassertions.md#locator-assertions-to-have-class) | Element has specified CSS class property | | [await test_timeouts#specify-expect-timeout-for-a-single-assertionrank 3 · score 0.770not in golden set ```js import { test, expect } from '@playwright/test'; test('example', async ({ page }) => { await expect(locator).toHaveText('hello', { timeout: 10_000 }); }); ``` Expected but not retrieved: Score across history 1 flip(s), all across config changes #1 0.79#2 0.79#3 0.87 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail degraded | to-global-timeoutIs there a timeout for the whole test run, and what is it by default? | timeouts | 0.862 | -0.055 | keyword_presence | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question Is there a timeout for the whole test run, and what is it by default? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: There is no default global timeout; set testConfig.globalTimeout. Actual answer Yes, Playwright Test supports a timeout for the whole test run, but there is no default global timeout [test_timeouts#global-timeout]. Attribution generationevery expected chunk was retrieved (recall 1.0) yet the case fails on keyword_presence. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 0.500 Retrieved chunks test_timeouts#advanced-low-level-timeoutsrank 1 · score 0.847not in golden set These are the low-level timeouts that are pre-configured by the test runner, you should not need to change these. If you happen to be in this section because your test are flaky, it is very likely that you should be looking for the solution elsewhere. |Timeout |Default |Description | |:----------|:----------------|:--------------------------------| |Action timeout| no timeout |Timeout for each action<br/><span style={{textTransform:'uppercase',fontSize:'smaller',fontWeight:'bold',opacity:'0.7'}}>Set in config</span><br/><code>{`{ use: { actionTimeout: 10_000 } }`}</code><br/><span style={{textTransform: 'uppercase',fontSize: 'smaller', fontWeight: 'bold', opacity: '0.7'}}>Override in test</span><br/>`locator.click({ timeout: 10_000 })` | |Navigation timeout| no timeout |Timeout for each navigation action<br/><span style={{textTransform:'uppercase',fontSize:'smaller',fontWeight:'bold',opacity:'0.7'}}>Set in config</span><br/><code>{`{ use: { navigationTimeout: 30_000 } }`}</code><br/><span style={{textTransform: 'uppercase',fontSize: 'smaller', fontWeight: 'bold', opacity: '0.7'}}>Override in test</span><br/>`page.goto('/', { timeout: 30_000 })` | |Global timeout|no timeout |Global timeout for the whole test run<br/><span style={{textTransform:'uppercase',fontSize:'smaller',fontWeight:'bold',opacity:'0.7'}}>Set in config</span><br/>`{ globalTimeout: 3_600_000 }`<br/> | |`beforeAll`/`afterAll` timeout|30_000 ms|Timeout for the hook<br/><span style={{textTransform:'uppercase',fontSize:'smaller',fontWeight:'bold',opacity:'0.7'}}>Set in hook</span><br/>`test.setTimeout(60_000)`<br/> | |Fixture timeout|no timeout |Timeout for an individual fixture<br/><span style={{textTransform:'uppercase',fontSize:'smaller',fontWeight:'bold',opacity:'0.7'}}>Set in fixture</span><br/>`{ scope: 'test', timeout: 30_000 }`<br/> | test_timeouts#global-timeoutrank 2 · score 0.795expected Playwright Test supports a timeout for the whole test run. This prevents excess resource usage when everything went wrong. There is no default global timeout, but you can set a reasonable one in the config, for example one hour. Global timeout produces the following error: ```txt Running 1000 tests using 10 workers 514 skipped 486 passed Timed out waiting 3600s for the entire test run ``` You can set global timeout in the config. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ globalTimeout: 3_600_000, }); ``` API reference: testConfig.globalTimeout. ci#global-timeoutrank 3 · score 0.782not in golden set Always set a [global timeout](./test-timeouts.md#global-timeout) in CI. By default a test run has no upper bound, so a suite that hangs, or that slowly grows past the job limit of your CI provider, is killed by the runner mid-run and does not produce the test report. With testConfig.globalTimeout set, Playwright stops the run itself. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Fail the run after an hour, so that the reporters still produce a report. globalTimeout: 60 * 60 * 1000, }); ``` The examples below therefore do not set a job-level timeout such as `timeout-minutes` in GitHub Actions. If you do add one, keep it comfortably above `globalTimeout`, so that Playwright always stops first. Score across history no pass/fail flips #1 0.91#2 0.92#3 0.86 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass degraded | to-beforeall-timeoutHow do I give a beforeAll hook more time than a normal test? | timeouts | 0.941 | -0.059 | - | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question How do I give a beforeAll hook more time than a normal test? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: test.setTimeout() / testInfo.setTimeout() inside the hook. Actual answer You can change the timeout for a `beforeAll` hook separately by calling `testInfo.setTimeout()` inside the hook [test_timeouts#change-timeout-for-beforeall-afterall-hook]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 1.000 Retrieved chunks test_timeouts#change-timeout-for-beforeall-afterall-hookrank 1 · score 0.849expected `beforeAll` and `afterAll` hooks have a separate timeout, by default equal to test timeout. You can change it separately for each hook by calling testInfo.setTimeout() inside the hook. ```js import { test, expect } from '@playwright/test'; test.beforeAll(async () => { // Set timeout for this hook. test.setTimeout(60000); }); ``` API reference: testInfo.setTimeout(). test_parameterize#before-and-after-hooksrank 2 · score 0.783not in golden set Most of the time you should put `beforeEach`, `beforeAll`, `afterEach` and `afterAll` hooks outside of `forEach`, so that hooks are executed just once: ```js test.beforeEach(async ({ page }) => { // ... }); test.afterEach(async ({ page }) => { // ... }); [ { name: 'Alice', expected: 'Hello, Alice!' }, { name: 'Bob', expected: 'Hello, Bob!' }, { name: 'Charlie', expected: 'Hello, Charlie!' }, ].forEach(({ name, expected }) => { test(`testing with ${name}`, async ({ page }) => { await page.goto(`https://example.com/greet?name=${name}`); await expect(page.getByRole('heading')).toHaveText(expected); }); }); ``` If you want to have hooks for each test, you can put them inside a `describe()` - so they are executed for each iteration / each individual test: ```js [ { name: 'Alice', expected: 'Hello, Alice!' }, { name: 'Bob', expected: 'Hello, Bob!' }, { name: 'Charlie', expected: 'Hello, Charlie!' }, ].forEach(({ name, expected }) => { test.describe(() => { test.beforeEach(async ({ page }) => { await page.goto(`https://example.com/greet?name=${name}`); }); test(`testing with ${expected}`, async ({ page }) => { await expect(page.getByRole('heading')).toHaveText(expected); }); }); }); ``` test_timeouts#change-timeout-from-a-beforeeach-hookrank 3 · score 0.777not in golden set ```js import { test, expect } from '@playwright/test'; test.beforeEach(async ({ page }, testInfo) => { // Extend timeout for all tests running this hook by 30 seconds. testInfo.setTimeout(testInfo.timeout + 30_000); }); ``` API reference: testInfo.setTimeout(). Score across history no pass/fail flips #1 1.00#2 1.00#3 0.94 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass degraded | to-fixture-timeoutCan a slow worker fixture have its own timeout instead of eating into the test timeout? | timeouts | 0.954 | -0.046 | - | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question Can a slow worker fixture have its own timeout instead of eating into the test timeout? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: Pass { timeout: N } in the fixture tuple. Actual answer Yes, you can set a separate, larger timeout for a slow worker-scoped fixture rather than sharing the overall test timeout [test_fixtures#fixture-timeout, test_timeouts#fixture-timeout]. You can change this timeout by passing the `timeout` option to the fixture [test_fixtures#worker-scoped-fixtures]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.667recall@3 1.000reciprocal_rank 1.000 Retrieved chunks test_fixtures#fixture-timeoutrank 1 · score 0.858expected Fixture is considered to be a part of a test, and so its setup and teardown running time counts towards the test timeout. Therefore, a slow fixture may cause test timeouts. You can set a separate larger timeout for such a fixture, and keep the overall test timeout small. ```js import { test as base, expect } from '@playwright/test'; const test = base.extend<{ slowFixture: string }>({ slowFixture: [async ({}, use) => { // ... perform a slow operation ... await use('hello'); }, { timeout: 60000 }] }); test('example test', async ({ slowFixture }) => { // ... }); ``` Unlike regular test-scoped fixtures, each [worker-scoped](#worker-scoped-fixtures) fixture has its own timeout, equal to the test timeout. You can change the timeout for a worker-scoped fixture in the same way. test_timeouts#fixture-timeoutrank 2 · score 0.854expected By default, [fixture](./test-fixtures) shares timeout with the test. However, for slow fixtures, especially [worker-scoped](./test-fixtures#worker-scoped-fixtures) ones, it is convenient to have a separate timeout. This way you can keep the overall test timeout small, and give the slow fixture more time. ```js import { test as base, expect } from '@playwright/test'; const test = base.extend<{ slowFixture: string }>({ slowFixture: [async ({}, use) => { // ... perform a slow operation ... await use('hello'); }, { timeout: 60_000 }] }); test('example test', async ({ slowFixture }) => { // ... }); ``` API reference: test.extend(). test_fixtures#worker-scoped-fixturesrank 3 · score 0.765not in golden set Playwright Test uses [worker processes](./test-parallel.md) to run test files. Similar to how test fixtures are set up for individual test runs, worker fixtures are set up for each worker process. That's where you can set up services, run servers, etc. Playwright Test will reuse the worker process for as many test files as it can, provided their worker fixtures match and hence environments are identical. Below we'll create an `account` fixture that will be shared by all tests in the same worker, and override the `page` fixture to log in to this account for each test. To generate unique accounts, we'll use the workerInfo.workerIndex that is available to any test or fixture. Note the tuple-like syntax for the worker fixture - we have to pass `{scope: 'worker'}` so that test runner sets this fixture up once per worker. In addition to only being run once per worker, worker-scoped fixtures also get a separate timeout equal to the default test timeout. You can change it by passing the `timeout` option. See [fixture timeout](#fixture-timeout) for more details. ```js import { test as Score across history no pass/fail flips #1 1.00#2 1.00#3 0.95 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass degraded | to-webserver-timeoutMy dev server takes ages to boot and Playwright gives up waiting for it. What do I change? | timeouts | 0.915 | -0.083 | - | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question My dev server takes ages to boot and Playwright gives up waiting for it. What do I change? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: webServer.timeout in the config. Actual answer You can increase the timeout by setting the `timeout` property within the `webServer` configuration in your Playwright config file [test_webserver#adding-a-server-timeout]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 1.000 Retrieved chunks test_webserver#adding-a-server-timeoutrank 1 · score 0.834expected Webservers can sometimes take longer to boot up. In this case, you can increase the timeout to wait for the server to start. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Rest of your config... // Run your local dev server before starting the tests webServer: { command: 'npm run start', url: 'http://localhost:3000', reuseExistingServer: !process.env.CI, timeout: 120 * 1000, }, }); ``` browsers#install-behind-a-firewall-or-a-proxy~2rank 2 · score 0.768not in golden set connection timeout; it limits how long the download may go without receiving data, and defaults to 30 seconds: ```bash PLAYWRIGHT_DOWNLOAD_CONNECTION_TIMEOUT=120000 npx playwright install ``` ```bash pip install playwright PLAYWRIGHT_DOWNLOAD_CONNECTION_TIMEOUT=120000 playwright install ``` ```bash PLAYWRIGHT_DOWNLOAD_CONNECTION_TIMEOUT=120000 mvn exec:java -e -D exec.mainClass=com.microsoft.playwright.CLI -D exec.args="install" ``` ```bash PLAYWRIGHT_DOWNLOAD_CONNECTION_TIMEOUT=120000 pwsh bin/Debug/netX/playwright.ps1 install ``` If you are [installing dependencies](#install-system-dependencies) and need to use a proxy on Linux, make sure to run the command as a root user. Otherwise, Playwright will attempt to become a root and will not pass environment variables like `HTTPS_PROXY` to the linux package manager. ```bash sudo HTTPS_PROXY=https://192.0.2.1 npx playwright install-deps ``` ```bash sudo HTTPS_PROXY=https://192.0.2.1 mvn exec:java -e -D exec.mainClass=com.microsoft.playwright.CLI -D exec.args="install-deps" ``` ```bash sudo HTTPS_PROXY=https://192.0.2.1 playwright install-deps ``` ```bash sudo HTTPS_PROXY=https://192.0.2.1 pwsh bin/Debug/netX/playwright.ps1 install-deps ``` test_webserver#configuring-a-web-serverrank 3 · score 0.759not in golden set Use the `webServer` property in your Playwright config to launch a development web server during the tests. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Run your local dev server before starting the tests webServer: { command: 'npm run start', url: 'http://localhost:3000', reuseExistingServer: !process.env.CI, stdout: 'ignore', stderr: 'pipe', }, }); ``` | Property | Description | | :- | :- | | testConfig.webServer | Launch a development web server (or multiple) during the tests. | | `command`| Shell command to start the local dev server of your app. | | `cwd` | Current working directory of the spawned process, defaults to the directory of the configuration file. | | `env` | Environment variables for the command. Defaults to inheriting `process.env` with `PLAYWRIGHT_TEST=1` added. | | `gracefulShutdown` | How to shut down the process. If unspecified, the process group is forcefully `SIGKILL`ed. If set to `{ signal: 'SIGTERM', timeout: 500 }`, the process group is sent a `SIGTERM` signal, followed by `SIGKILL` if it doesn't exit within 500ms. You can also use `SIGINT` as the signal instead. A Score across history no pass/fail flips #1 1.00#2 1.00#3 0.92 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail regressed | par-retries-flagHow do I make Playwright retry failing tests? | parallel | 0.902 | -0.096 | grounding | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question How do I make Playwright retry failing tests? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: --retries=N on the CLI or retries in the config. Actual answer You can enable test retries in Playwright by passing the `--retries` command-line flag or by configuring the `retries` option in your configuration file [test_retries#retries]. Attribution generationevery expected chunk was retrieved (recall 1.0) yet the case fails on grounding. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 1.000 Retrieved chunks test_retries#retriesrank 1 · score 0.862expected Playwright supports **test retries**. When enabled, failing tests will be retried multiple times until they pass, or until the maximum number of retries is reached. By default failing tests are not retried. ```bash npx playwright test --retries=3 ``` You can configure retries in the configuration file: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Give failing tests 3 retry attempts retries: 3, }); ``` Playwright Test will categorize tests as follows: - "passed" - tests that passed on the first run; - "flaky" - tests that failed on the first run, but passed when retried; - "failed" - tests that failed on the first run and failed all retries. ```bash Running 3 tests using 1 worker ✓ example.spec.ts:4:2 › first passes (438ms) x example.spec.ts:5:2 › second flaky (691ms) ✓ example.spec.ts:5:2 › second flaky (522ms) ✓ example.spec.ts:6:2 › third passes (932ms) 1 flaky example.spec.ts:5:2 › second flaky 2 passed (4s) ``` You can detect retries at runtime with testInfo.retry, which is accessible to any test, hook or fixture. Here is an example that clears some server-side state before test_parallel#limit-failures-and-fail-fastrank 2 · score 0.841not in golden set You can limit the number of failed tests in the whole test suite by setting `maxFailures` config option or passing `--max-failures` command line flag. When running with "max failures" set, Playwright Test will stop after reaching this number of failed tests and skip any tests that were not executed yet. This is useful to avoid wasting resources on broken test suites. Passing command line option: ```bash npx playwright test --max-failures=10 ``` Setting in the configuration file: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Limit the number of failures on CI to save resources maxFailures: process.env.CI ? 10 : undefined, }); ``` ci_intro#viewing-the-tracerank 3 · score 0.822not in golden set Once you have served the report using `npx playwright show-report`, click on the trace icon next to the test's file name as seen in the image above. You can then view the trace of your tests and inspect each action to try to find out why the tests are failing. <img src="https://github.com/microsoft/playwright/assets/13063165/10fe3585-8401-4051-b1c2-b2e92ac4c274" alt="playwright trace viewer" width="3598" height="2218" /> Score across history 1 flip(s), all across config changes #1 1.00#2 1.00#3 0.90 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass degraded | par-flaky-definitionWhat does Playwright count as a "flaky" test in its results? | parallel | 0.929 | -0.069 | - | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question What does Playwright count as a "flaky" test in its results? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: Exact definition from the retries guide. Actual answer Playwright categorizes a test as "flaky" if it failed on the first run, but passed when retried [test_retries#retries]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 1.000 Retrieved chunks test_retries#retriesrank 1 · score 0.778expected Playwright supports **test retries**. When enabled, failing tests will be retried multiple times until they pass, or until the maximum number of retries is reached. By default failing tests are not retried. ```bash npx playwright test --retries=3 ``` You can configure retries in the configuration file: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Give failing tests 3 retry attempts retries: 3, }); ``` Playwright Test will categorize tests as follows: - "passed" - tests that passed on the first run; - "flaky" - tests that failed on the first run, but passed when retried; - "failed" - tests that failed on the first run and failed all retries. ```bash Running 3 tests using 1 worker ✓ example.spec.ts:4:2 › first passes (438ms) x example.spec.ts:5:2 › second flaky (691ms) ✓ example.spec.ts:5:2 › second flaky (522ms) ✓ example.spec.ts:6:2 › third passes (932ms) 1 flaky example.spec.ts:5:2 › second flaky 2 passed (4s) ``` You can detect retries at runtime with testInfo.retry, which is accessible to any test, hook or fixture. Here is an example that clears some server-side state before writing_tests#assertionsrank 2 · score 0.770not in golden set Playwright includes [test assertions](./test-assertions.md) in the form of `expect` function. To make an assertion, call `expect(value)` and choose a matcher that reflects the expectation. Playwright includes async matchers that wait until the expected condition is met. Using these matchers makes tests non-flaky and resilient. For example, this code waits until the page gets the title containing "Playwright": ```js await expect(page).toHaveTitle(/Playwright/); ``` Here are the most popular async assertions. For the complete list, see [assertions guide](./test-assertions.md): | Assertion | Description | | :- | :- | | locatorAssertions.toBeChecked() | Checkbox is checked | | locatorAssertions.toBeEnabled() | Control is enabled | | locatorAssertions.toBeVisible() | Element is visible | | locatorAssertions.toContainText() | Element contains text | | locatorAssertions.toHaveAttribute() | Element has attribute | | locatorAssertions.toHaveCount() | List of elements has given length | | locatorAssertions.toHaveText() | Element matches text | | locatorAssertions.toHaveValue() | Input element has value | | pageAssertions.toHaveTitle() | Page has title | | pageAssertions.toHaveURL() | Page has URL | Playwright also includes generic matchers like `toEqual`, `toContain`, `toBeTruthy` that can be used to assert any conditions. These assertions do not test_parallel#avoiding-shared-state-in-parallel-testsrank 3 · score 0.756not in golden set Playwright runs tests in separate worker processes, each with its own isolated BrowserContext, so cookies, storage and in-memory globals are already isolated. Flakiness comes from state that lives *outside* a single test. Here are recipes for the common cases. Score across history 1 flip(s), all across config changes #1 0.67#2 1.00#3 0.93 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail degraded | par-retries-defaultAre failing tests retried by default? | parallel | 0.038 | -0.474 | exact_matchfuzzy_matchkeyword_presencerefusal_behaviourexpected_citation_overlap | 0.0 | |||||||||||||||||||||||||||||||||||||||||
Question Are failing tests retried by default? Expected No. By default failing tests are not retried. required: Actual answer I do not have enough information in the provided documents to answer that. Attribution retrievalno expected chunk was retrieved (hit rate 0), so the generator never saw the evidence. Checks
Retrieval metrics hit_rate@3 0.000precision@3 0.000recall@3 0.000reciprocal_rank 0.000 Retrieved chunks test_retries#introductionrank 1 · score 0.759not in golden set Test retries are a way to automatically re-run a test when it fails. This is useful when a test is flaky and fails intermittently. Test retries are configured in the [configuration file](./test-configuration.md). browser_contexts#why-is-test-isolation-importantrank 2 · score 0.737not in golden set - No failure carry-over. If one test fails it doesn't affect the other test. - Easy to debug errors or flakiness, because you can run just a single test as many times as you'd like. - Don't have to think about the order when running in parallel, sharding, etc. test_retries#failures~2rank 3 · score 0.737not in golden set with the browser and will start a new one. Testing will continue in the new worker process starting with the next test. * Worker process #1 starts * `beforeAll` hook runs * `first good` passes * `second flaky` fails * `afterAll` hook runs * Worker process #2 starts * `beforeAll` hook runs again * `third good` passes * `afterAll` hook runs If you **enable [retries](#retries)**, second worker process will start by retrying the failed test and continue from there. * Worker process #1 starts * `beforeAll` hook runs * `first good` passes * `second flaky` fails * `afterAll` hook runs * Worker process #2 starts * `beforeAll` hook runs again * `second flaky` is retried and passes * `third good` passes * `afterAll` hook runs This scheme works perfectly for independent tests and guarantees that failing tests can't affect healthy ones. Expected but not retrieved: Score across history 1 flip(s), all across config changes #1 0.76#2 0.51#3 0.04 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass degraded | par-workers-cliHow do I limit the number of parallel workers on CI but keep the default locally? | parallel | 0.948 | -0.045 | - | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question How do I limit the number of parallel workers on CI but keep the default locally? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: workers: process.env.CI ? 2 : undefined. Actual answer You can limit the number of workers on CI while using the default locally by setting `workers: process.env.CI ? 2 : undefined` (or `1` for CI) in your configuration file [test_parallel#limit-workers, ci#workers]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 1.000 Retrieved chunks test_parallel#limit-workersrank 1 · score 0.867expected You can control the maximum number of parallel worker processes via [command line](./test-cli.md) or in the [configuration file](./test-configuration.md). From the command line: ```bash npx playwright test --workers 4 ``` In the configuration file: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Limit the number of workers on CI, use default locally workers: process.env.CI ? 2 : undefined, }); ``` ci#workersrank 2 · score 0.796not in golden set We recommend setting [workers](./api/class-testconfig.md#test-config-workers) to "1" in CI environments to prioritize stability and reproducibility. Running tests sequentially ensures each test gets the full system resources, avoiding potential conflicts. However, if you have a powerful self-hosted CI system, you may enable [parallel](./test-parallel.md) tests. For wider parallelization, consider [sharding](./test-parallel.md#shard-tests-between-multiple-machines) - distributing tests across multiple CI jobs. ```js import { defineConfig, devices } from '@playwright/test'; export default defineConfig({ // Opt out of parallel tests on CI. workers: process.env.CI ? 1 : undefined, }); ``` test_parallel#disable-parallelismrank 3 · score 0.791not in golden set You can disable any parallelism by allowing just a single worker at any time. Either set `workers: 1` option in the configuration file or pass `--workers=1` to the command line. ```bash npx playwright test --workers=1 ``` Score across history no pass/fail flips #1 1.00#2 0.99#3 0.95 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass improved | par-disable-parallelHow do I turn parallelism off completely? | parallel | 0.935 | +0.102 | - | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question How do I turn parallelism off completely? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Actual answer You can disable parallelism completely by setting the `workers: 1` option in the configuration file or by passing `--workers=1` to the command line [test_parallel#disable-parallelism]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 1.000 Retrieved chunks test_parallel#disable-parallelismrank 1 · score 0.801expected You can disable any parallelism by allowing just a single worker at any time. Either set `workers: 1` option in the configuration file or pass `--workers=1` to the command line. ```bash npx playwright test --workers=1 ``` test_parallel#limit-workersrank 2 · score 0.699not in golden set You can control the maximum number of parallel worker processes via [command line](./test-cli.md) or in the [configuration file](./test-configuration.md). From the command line: ```bash npx playwright test --workers 4 ``` In the configuration file: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Limit the number of workers on CI, use default locally workers: process.env.CI ? 2 : undefined, }); ``` test_parallel#sort-test-files-alphabeticallyrank 3 · score 0.693not in golden set When you **disable parallel test execution**, Playwright Test runs test files in alphabetical order. You can use some naming convention to control the test order, for example `001-user-signin-flow.spec.ts`, `002-create-new-document.spec.ts` and so on. Score across history 1 flip(s), all across config changes #1 0.66#2 0.83#3 0.93 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail degraded | par-serial-modeMy tests depend on each other. How do I make them run in order and stop after the first failure? | parallel | 0.526 | -0.140 | keyword_presenceexpected_citation_overlap | 0.0 | |||||||||||||||||||||||||||||||||||||||||
Question My tests depend on each other. How do I make them run in order and stop after the first failure? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: test.describe.configure({ mode: 'serial' }); subsequent tests are skipped on failure. Actual answer To run test files in order, you must disable parallel test execution so that Playwright Test runs them alphabetically [test_parallel#sort-test-files-alphabetically]. To stop the test suite after failures, you can set the `maxFailures` config option or pass the `--max-failures` command-line flag [test_parallel#limit-failures-and-fail-fast]. Attribution retrievalno expected chunk was retrieved (hit rate 0), so the generator never saw the evidence. Checks
Retrieval metrics hit_rate@3 0.000precision@3 0.000recall@3 0.000reciprocal_rank 0.000 Retrieved chunks test_parallel#sort-test-files-alphabeticallyrank 1 · score 0.784not in golden set When you **disable parallel test execution**, Playwright Test runs test files in alphabetical order. You can use some naming convention to control the test order, for example `001-user-signin-flow.spec.ts`, `002-create-new-document.spec.ts` and so on. test_parallel#parallelize-tests-in-a-single-filerank 2 · score 0.775not in golden set By default, tests in a single file are run in order. If you have many independent tests in a single file, you might want to run them in parallel with test.describe.configure(). Note that parallel tests are executed in separate worker processes and cannot share any state or global variables. Each test executes all relevant hooks just for itself, including `beforeAll` and `afterAll`. ```js import { test } from '@playwright/test'; test.describe.configure({ mode: 'parallel' }); test('runs in parallel 1', async ({ page }) => { /* ... */ }); test('runs in parallel 2', async ({ page }) => { /* ... */ }); ``` Alternatively, you can opt-in all tests into this fully-parallel mode in the configuration file: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ fullyParallel: true, }); ``` You can also opt in for fully-parallel mode for just a few projects: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // runs all tests in all files of a specific project in parallel projects: [ { name: 'chromium', use: { ...devices['Desktop Chrome'] }, fullyParallel: true, }, ] test_parallel#limit-failures-and-fail-fastrank 3 · score 0.766not in golden set You can limit the number of failed tests in the whole test suite by setting `maxFailures` config option or passing `--max-failures` command line flag. When running with "max failures" set, Playwright Test will stop after reaching this number of failed tests and skip any tests that were not executed yet. This is useful to avoid wasting resources on broken test suites. Passing command line option: ```bash npx playwright test --max-failures=10 ``` Setting in the configuration file: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Limit the number of failures on CI to save resources maxFailures: process.env.CI ? 10 : undefined, }); ``` Expected but not retrieved: Score across history no pass/fail flips #1 0.66#2 0.67#3 0.53 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass degraded | par-max-failuresCan I stop the whole run after ten failures so I don't waste CI minutes? | parallel | 0.951 | -0.049 | - | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question Can I stop the whole run after ten failures so I don't waste CI minutes? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: --max-failures=10 or maxFailures in the config. Actual answer Yes, you can limit the number of failed tests by setting the `maxFailures` config option or passing the `--max-failures` command line flag [test_parallel#limit-failures-and-fail-fast]. For example, you can set it to 10 to stop the test suite after reaching that number of failures [test_parallel#limit-failures-and-fail-fast]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 0.333 Retrieved chunks test_use_options#trace-modes~2rank 1 · score 0.766not in golden set on first run | Fails, then passes on retry | Fails on every run | | :- | :- | :- | :- | | `'off'` | — | — | — | | `'on'` | first run | first run + retry | all three runs | | `'retain-on-failure'` | — | first run | all three runs | | `'retain-on-first-failure'` | — | first run | first run | | `'retain-on-failure-and-retries'` | — | first run + retry | all three runs | | `'on-first-retry'` | — | first retry | first retry | | `'on-all-retries'` | — | first retry | both retries | test_use_options#video-modes~2rank 2 · score 0.748not in golden set :- | :- | :- | | `'off'` | — | — | — | | `'on'` | first run | first run + retry | all three runs | | `'retain-on-failure'` | — | first run | all three runs | | `'retain-on-first-failure'` | — | first run | first run | | `'retain-on-failure-and-retries'` | — | first run + retry | all three runs | | `'on-first-retry'` | — | first retry | first retry | | `'on-all-retries'` | — | first retry | both retries | test_parallel#limit-failures-and-fail-fastrank 3 · score 0.726expected You can limit the number of failed tests in the whole test suite by setting `maxFailures` config option or passing `--max-failures` command line flag. When running with "max failures" set, Playwright Test will stop after reaching this number of failed tests and skip any tests that were not executed yet. This is useful to avoid wasting resources on broken test suites. Passing command line option: ```bash npx playwright test --max-failures=10 ``` Setting in the configuration file: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Limit the number of failures on CI to save resources maxFailures: process.env.CI ? 10 : undefined, }); ``` Score across history no pass/fail flips #1 1.00#2 1.00#3 0.95 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail regressed | par-shard-flagHow do I split my suite across four CI machines? | parallel | 0.000 | -0.825 | keyword_presencerefusal_behaviourexpected_citation_overlap | 0.0 | |||||||||||||||||||||||||||||||||||||||||
Question How do I split my suite across four CI machines? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: --shard=1/4 ... --shard=4/4. Actual answer I do not have enough information in the provided documents to answer that. Attribution retrievalno expected chunk was retrieved (hit rate 0), so the generator never saw the evidence. Checks
Retrieval metrics hit_rate@3 0.000precision@3 0.000recall@3 0.000reciprocal_rank 0.000 Retrieved chunks ci#ci-configurationsrank 1 · score 0.690not in golden set The [Command line tools](./browsers#install-system-dependencies) can be used to install all operating system dependencies in CI. test_parallel#limit-workersrank 2 · score 0.675not in golden set You can control the maximum number of parallel worker processes via [command line](./test-cli.md) or in the [configuration file](./test-configuration.md). From the command line: ```bash npx playwright test --workers 4 ``` In the configuration file: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Limit the number of workers on CI, use default locally workers: process.env.CI ? 2 : undefined, }); ``` ci#workersrank 3 · score 0.663not in golden set We recommend setting [workers](./api/class-testconfig.md#test-config-workers) to "1" in CI environments to prioritize stability and reproducibility. Running tests sequentially ensures each test gets the full system resources, avoiding potential conflicts. However, if you have a powerful self-hosted CI system, you may enable [parallel](./test-parallel.md) tests. For wider parallelization, consider [sharding](./test-parallel.md#shard-tests-between-multiple-machines) - distributing tests across multiple CI jobs. ```js import { defineConfig, devices } from '@playwright/test'; export default defineConfig({ // Opt out of parallel tests on CI. workers: process.env.CI ? 1 : undefined, }); ``` Expected but not retrieved: Score across history 1 flip(s), all across config changes #1 1.00#2 0.82#3 0.00 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass degraded | par-merge-reportsAfter sharding, how do I get one HTML report instead of four? | parallel | 0.857 | -0.057 | - | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question After sharding, how do I get one HTML report instead of four? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: npx playwright merge-reports with blob reports. Actual answer To get one HTML report, put the blob report files from all shards into a single directory and run `npx playwright merge-reports --reporter html ./all-blob-reports` [test_sharding#merging-reports-from-multiple-shards]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 0.500reciprocal_rank 1.000 Retrieved chunks test_sharding#merging-reports-from-multiple-shardsrank 1 · score 0.762expected In the previous example, each test shard has its own test report. If you want to have a combined report showing all the test results from all the shards, you can merge them. Start with adding `blob` reporter to the config when running on CI: ```js export default defineConfig({ testDir: './tests', reporter: process.env.CI ? 'blob' : 'html', }); ``` Blob report contains information about all the tests that were run and their results as well as all test attachments such as traces and screenshot diffs. Blob reports can be merged and converted to any other Playwright report. By default, blob report will be generated into `blob-report` directory. You can learn about [blob report options here](./test-reporters.md#blob-reporter). To merge reports from multiple shards, put the blob report files into a single directory, for example `all-blob-reports`. Blob report names contain shard number, so they will not clash. Afterwards, run `npx playwright merge-reports` command: ```bash npx playwright merge-reports --reporter html ./all-blob-reports ``` This will produce a standard HTML report into `playwright-report` directory. test_cli#show-reportrank 2 · score 0.751not in golden set Display HTML report from previous test run. [Read more about the HTML reporter](./test-reporters#html-reporter). test_reporters#blob-reporter~2rank 3 · score 0.745not in golden set blob report to merge multiple shards, you don't have to pass any options. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ reporter: 'blob', }); ``` </TabItem> <TabItem value="environments"> When running tests in different environments, you might want to use testConfig.tag to add a global tag corresponding to the environment. This tag will bring clarity to the merged report, and it will be used to produce a unique blob report name. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ reporter: 'blob', tag: process.env.CI_ENVIRONMENT_NAME, // for example "@APIv2" or "@linux" }); ``` </TabItem> </Tabs> Blob report supports following configuration options and environment variables: | Environment Variable Name | Reporter Config Option| Description | Default |---|---|---|---| | `PLAYWRIGHT_BLOB_OUTPUT_DIR` | `outputDir` | Directory to save the output. Existing content is deleted before writing the new report. | `blob-report` | `PLAYWRIGHT_BLOB_OUTPUT_NAME` | `fileName` | Report file name. | `report-<hash>-<shard_number>.zip` | `PLAYWRIGHT_BLOB_OUTPUT_FILE` | `outputFile` | Full path to the output file. If defined, `outputDir` and `fileName` will be ignored. | `undefined` Expected but not retrieved: Score across history 1 flip(s), all across config changes #1 0.67#2 0.91#3 0.86 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass degraded | par-worker-indexHow can a test find out which worker it is running in, so each worker uses its own test account? | parallel | 0.814 | -0.101 | - | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question How can a test find out which worker it is running in, so each worker uses its own test account? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: testInfo.workerIndex / process.env.TEST_WORKER_INDEX. The parallelIndex distinction is a bonus. Actual answer To differentiate between workers and generate unique accounts, tests and fixtures can use `testInfo.workerIndex` or `workerInfo.workerIndex` [test_parallel#isolate-test-data-between-parallel-workers, test_fixtures#worker-scoped-fixtures]. Additionally, `test.info().parallelIndex` can be used as a unique identifier for a worker [auth#authenticate-with-api-request~2]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 0.500reciprocal_rank 1.000 Retrieved chunks test_parallel#isolate-test-data-between-parallel-workersrank 1 · score 0.787expected You can leverage `process.env.TEST_WORKER_INDEX` or testInfo.workerIndex mentioned above to isolate user data in the database between tests running on different workers. All tests run by the worker reuse the same user. Create `playwright/fixtures.ts` file that will [create `dbUserName` fixture](./test-fixtures#creating-a-fixture) and initialize a new user in the test database. Use testInfo.workerIndex to differentiate between workers. ```js import { test as baseTest, expect } from '@playwright/test'; // Import project utils for managing users in the test database. import { createUserInTestDatabase, deleteUserFromTestDatabase } from './my-db-utils'; export * from '@playwright/test'; export const test = baseTest.extend<{}, { dbUserName: string }>({ // Returns db user name unique for the worker. dbUserName: [async ({ }, use) => { // Use workerIndex as a unique identifier for each worker. const userName = `user-${test.info().workerIndex}`; // Initialize user in the database. await createUserInTestDatabase(userName); await use(userName); // Clean up after the tests are done. await deleteUserFromTestDatabase(userName); }, { scope: 'worker' }], }); ``` Now, each test file should import `test` from our fixtures file instead of `@playwright/test`. ```js // Important: import our fixtures. import { test, expect } from '../playwright/fixtures'; test('test', auth#authenticate-with-api-request~2rank 2 · score 0.763not in golden set workerStorageState: [async ({}, use) => { // Use parallelIndex as a unique identifier for each worker. const id = test.info().parallelIndex; const fileName = path.resolve(test.info().project.outputDir, `.auth/${id}.json`); if (fs.existsSync(fileName)) { // Reuse existing authentication state if any. await use(fileName); return; } // Important: make sure we authenticate in a clean environment by unsetting storage state. const context = await request.newContext({ storageState: undefined }); // Acquire a unique account, for example create a new one. // Alternatively, you can have a list of precreated accounts for testing. // Make sure that accounts are unique, so that multiple team members // can run tests at the same time without interference. const account = await acquireAccount(id); // Send authentication request. Replace with your own. await context.post('https://github.com/login', { form: { 'user': 'user', 'password': 'password' } }); await context.storageState({ path: fileName }); await context.dispose(); await use(fileName); }, { scope: 'worker' }], }); ``` test_fixtures#worker-scoped-fixturesrank 3 · score 0.757not in golden set Playwright Test uses [worker processes](./test-parallel.md) to run test files. Similar to how test fixtures are set up for individual test runs, worker fixtures are set up for each worker process. That's where you can set up services, run servers, etc. Playwright Test will reuse the worker process for as many test files as it can, provided their worker fixtures match and hence environments are identical. Below we'll create an `account` fixture that will be shared by all tests in the same worker, and override the `page` fixture to log in to this account for each test. To generate unique accounts, we'll use the workerInfo.workerIndex that is available to any test or fixture. Note the tuple-like syntax for the worker fixture - we have to pass `{scope: 'worker'}` so that test runner sets this fixture up once per worker. In addition to only being run once per worker, worker-scoped fixtures also get a separate timeout equal to the default test timeout. You can change it by passing the `timeout` option. See [fixture timeout](#fixture-timeout) for more details. ```js import { test as Expected but not retrieved: Score across history 1 flip(s), all across config changes #1 0.66#2 0.91#3 0.81 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail degraded | par-isolation-contextHow does Playwright keep one test's cookies and local storage from leaking into the next test? | parallel | 0.557 | -0.110 | keyword_presenceexpected_citation_overlap | 0.0 | |||||||||||||||||||||||||||||||||||||||||
Question How does Playwright keep one test's cookies and local storage from leaking into the next test? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: A fresh browser context per test. Actual answer Playwright runs tests in separate worker processes, each equipped with its own isolated BrowserContext to isolate cookies, storage, and in-memory globals [test_parallel#avoiding-shared-state-in-parallel-tests]. Attribution retrievalno expected chunk was retrieved (hit rate 0), so the generator never saw the evidence. Checks
Retrieval metrics hit_rate@3 0.000precision@3 0.000recall@3 0.000reciprocal_rank 0.000 Retrieved chunks auth#avoid-authentication-in-some-testsrank 1 · score 0.855not in golden set You can reset storage state in a test file to avoid authentication that was set up for the whole project. ```js import { test } from '@playwright/test'; // Reset storage state for this file to avoid being authenticated test.use({ storageState: { cookies: [], origins: [] } }); test('not signed in test', async ({ page }) => { // ... }); ``` test_parallel#avoiding-shared-state-in-parallel-testsrank 2 · score 0.803not in golden set Playwright runs tests in separate worker processes, each with its own isolated BrowserContext, so cookies, storage and in-memory globals are already isolated. Flakiness comes from state that lives *outside* a single test. Here are recipes for the common cases. test_global_setup_teardown#examplerank 3 · score 0.797not in golden set Here is a global setup example that authenticates once and reuses authentication state in tests. It uses the `baseURL` and `storageState` options from the configuration file. ```js import { chromium, type FullConfig } from '@playwright/test'; async function globalSetup(config: FullConfig) { const { baseURL, storageState } = config.projects[0].use; const browser = await chromium.launch(); const page = await browser.newPage(); await page.goto(baseURL!); await page.getByLabel('User Name').fill('user'); await page.getByLabel('Password').fill('password'); await page.getByText('Sign in').click(); await page.context().storageState({ path: storageState as string }); await browser.close(); } export default globalSetup; ``` Specify `globalSetup`, `baseURL` and `storageState` in the configuration file. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ globalSetup: require.resolve('./global-setup'), use: { baseURL: 'http://localhost:3000/', storageState: 'state.json', }, }); ``` Tests start already authenticated because we specify `storageState` that was populated by global setup. ```js import { test } from '@playwright/test'; test('test', async ({ page }) => { await page.goto('/'); // You are signed in! }); ``` You can make arbitrary data available in your tests from your global setup file by setting them as environment variables via `process.env`. ```js import type { FullConfig } from '@playwright/test'; async function Expected but not retrieved: Score across history no pass/fail flips #1 0.66#2 0.67#3 0.56 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail degraded | cfg-worker-fixtureHow do I set something up once per worker instead of once per test? | config | 0.000 | -0.654 | keyword_presencerefusal_behaviourexpected_citation_overlap | 0.0 | |||||||||||||||||||||||||||||||||||||||||
Question How do I set something up once per worker instead of once per test? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: The tuple syntax with { scope: 'worker' }. Actual answer I do not have enough information in the provided documents to answer that. Attribution retrievalno expected chunk was retrieved (hit rate 0), so the generator never saw the evidence. Checks
Retrieval metrics hit_rate@3 0.000precision@3 0.000recall@3 0.000reciprocal_rank 0.000 Retrieved chunks ci#workersrank 1 · score 0.826not in golden set We recommend setting [workers](./api/class-testconfig.md#test-config-workers) to "1" in CI environments to prioritize stability and reproducibility. Running tests sequentially ensures each test gets the full system resources, avoiding potential conflicts. However, if you have a powerful self-hosted CI system, you may enable [parallel](./test-parallel.md) tests. For wider parallelization, consider [sharding](./test-parallel.md#shard-tests-between-multiple-machines) - distributing tests across multiple CI jobs. ```js import { defineConfig, devices } from '@playwright/test'; export default defineConfig({ // Opt out of parallel tests on CI. workers: process.env.CI ? 1 : undefined, }); ``` test_parallel#limit-workersrank 2 · score 0.797not in golden set You can control the maximum number of parallel worker processes via [command line](./test-cli.md) or in the [configuration file](./test-configuration.md). From the command line: ```bash npx playwright test --workers 4 ``` In the configuration file: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Limit the number of workers on CI, use default locally workers: process.env.CI ? 2 : undefined, }); ``` getting_started_vscode#global-setuprank 3 · score 0.782not in golden set For tasks that need to run only once before all tests (like seeding a database), use **Global Setup**. You can trigger the global setup and teardown manually from the Playwright sidebar.  Expected but not retrieved: Score across history 1 flip(s), all across config changes #1 0.99#2 0.65#3 0.00 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass degraded | cfg-auto-fixtureCan a fixture run for every test without each test having to list it as a parameter? | config | 0.725 | -0.109 | expected_citation_overlap | 0.0 | |||||||||||||||||||||||||||||||||||||||||
Question Can a fixture run for every test without each test having to list it as a parameter? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Actual answer Yes, you can declare them as auto fixtures with `scope: 'worker'` and `auto: true` so they automatically run for every worker without being redeclared in every file [test_fixtures#adding-global-beforeall-afterall-hooks]. Checks
Retrieval metrics hit_rate@3 0.000precision@3 0.000recall@3 0.000reciprocal_rank 0.000 Retrieved chunks test_fixtures#combine-custom-fixtures-from-multiple-modulesrank 1 · score 0.784not in golden set You can merge test fixtures from multiple files or modules: ```js import { mergeTests } from '@playwright/test'; import { test as dbTest } from 'database-test-utils'; import { test as a11yTest } from 'a11y-test-utils'; export const test = mergeTests(dbTest, a11yTest); ``` ```js import { test } from './fixtures'; test('passes', async ({ database, page, a11y }) => { // use database and a11y fixtures. }); ``` test_fixtures#using-a-fixturerank 2 · score 0.783not in golden set Just mention a fixture in your test function argument, and the test runner will take care of it. Fixtures are also available in hooks and other fixtures. If you use TypeScript, fixtures will be type safe. Below we use the `todoPage` and `settingsPage` fixtures that we defined above. ```js import { test, expect } from './my-test'; test.beforeEach(async ({ settingsPage }) => { await settingsPage.switchToDarkMode(); }); test('basic test', async ({ todoPage, page }) => { await todoPage.addToDo('something nice'); await expect(page.getByTestId('todo-title')).toContainText(['something nice']); }); ``` test_fixtures#adding-global-beforeall-afterall-hooksrank 3 · score 0.777not in golden set test.beforeAll() and test.afterAll() hooks run before/after all tests declared in the same file and same test.describe() block (if any), once per worker process. If you want to declare hooks that run before/after all tests in every file, you can declare them as auto fixtures with `scope: 'worker'` as follows: ```js import { test as base } from '@playwright/test'; export const test = base.extend<{}, { forEachWorker: void }>({ forEachWorker: [async ({}, use) => { // This code runs before all the tests in the worker process. console.log(`Starting test worker ${test.info().workerIndex}`); await use(); // This code runs after all the tests in the worker process. console.log(`Stopping test worker ${test.info().workerIndex}`); }, { scope: 'worker', auto: true }], // automatically starts for every worker. }); ``` And then import the fixtures in all your tests: ```js import { test } from './fixtures'; import { expect } from '@playwright/test'; test('basic', async ({ }) => { // ... }); ``` Note that the fixtures will still run once per [worker process](./test-parallel.md#worker-processes), but you don't need to redeclare them in every file. Expected but not retrieved: Score across history 1 flip(s), all across config changes #1 0.67#2 0.83#3 0.72 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass degraded | cfg-fixture-orderIf fixture A depends on fixture B, which one is torn down first? | config | 0.927 | -0.070 | - | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question If fixture A depends on fixture B, which one is torn down first? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: hard - B is set up before A and torn down after A; wording in the chunk is "B is always set up before A and torn down after A". Actual answer Fixture A is torn down first, while fixture B is torn down after A [test_fixtures#execution-order]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 1.000 Retrieved chunks test_fixtures#execution-orderrank 1 · score 0.760expected Each fixture has a setup and teardown phase before and after the `await use()` call in the fixture. Setup is executed before the test/hook requiring it is run, and teardown is executed when the fixture is no longer being used by the test/hook. Fixtures follow these rules to determine the execution order: * When fixture A depends on fixture B: B is always set up before A and torn down after A. * Non-automatic fixtures are executed lazily, only when the test/hook needs them. * Test-scoped fixtures are torn down after each test, while worker-scoped fixtures are only torn down when the worker process executing tests is torn down. Consider the following example: ```js import { test as base } from '@playwright/test'; const test = base.extend<{ testFixture: string, autoTestFixture: string, unusedFixture: string, }, { workerFixture: string, autoWorkerFixture: string, }>({ workerFixture: [async ({ browser }) => { // workerFixture setup... await use('workerFixture'); // workerFixture teardown... }, { scope: 'worker' }], autoWorkerFixture: [async ({ browser }) => { // autoWorkerFixture setup... await use('autoWorkerFixture'); // autoWorkerFixture teardown... }, { scope: 'worker', auto: true test_fixtures#with-fixturesrank 2 · score 0.714not in golden set Fixtures have a number of advantages over before/after hooks: - Fixtures **encapsulate** setup and teardown in the same place so it is easier to write. So if you have an after hook that tears down what was created in a before hook, consider turning them into a fixture. - Fixtures are **reusable** between test files - you can define them once and use them in all your tests. That's how Playwright's built-in `page` fixture works. So if you have a helper function that is used in multiple tests, consider turning it into a fixture. - Fixtures are **on-demand** - you can define as many fixtures as you'd like, and Playwright Test will setup only the ones needed by your test and nothing else. - Fixtures are **composable** - they can depend on each other to provide complex behaviors. - Fixtures are **flexible**. Tests can use any combination of fixtures to precisely tailor the environment to their needs, without affecting other tests. - Fixtures simplify **grouping**. You no longer need to wrap tests in `describe`s that set up their environment, and test_fixtures#fixture-timeoutrank 3 · score 0.677not in golden set Fixture is considered to be a part of a test, and so its setup and teardown running time counts towards the test timeout. Therefore, a slow fixture may cause test timeouts. You can set a separate larger timeout for such a fixture, and keep the overall test timeout small. ```js import { test as base, expect } from '@playwright/test'; const test = base.extend<{ slowFixture: string }>({ slowFixture: [async ({}, use) => { // ... perform a slow operation ... await use('hello'); }, { timeout: 60000 }] }); test('example test', async ({ slowFixture }) => { // ... }); ``` Unlike regular test-scoped fixtures, each [worker-scoped](#worker-scoped-fixtures) fixture has its own timeout, equal to the test timeout. You can change the timeout for a worker-scoped fixture in the same way. Score across history no pass/fail flips #1 1.00#2 1.00#3 0.93 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail regressed | cfg-project-dependenciesHow do I make a login setup step run before my chromium and firefox projects? | config | 0.501 | -0.325 | keyword_presenceexpected_citation_overlapgrounding | 0.0 | |||||||||||||||||||||||||||||||||||||||||
Question How do I make a login setup step run before my chromium and firefox projects? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: A setup project listed in the other projects' dependencies. Actual answer You can create a new `setup` project in your config and declare it as a dependency for your chromium and firefox testing projects [auth#basic-shared-account-in-all-tests~2]. Additionally, configure the setup project with a `testMatch` pattern (such as `/.*\.setup\.ts/`) so that it always runs before all the tests [auth#basic-shared-account-in-all-tests~2, auth#basic-shared-account-in-all-tests~3]. Attribution retrievalno expected chunk was retrieved (hit rate 0), so the generator never saw the evidence. Checks
Retrieval metrics hit_rate@3 0.000precision@3 0.000recall@3 0.000reciprocal_rank 0.000 Retrieved chunks auth#basic-shared-account-in-all-tests~3rank 1 · score 0.797not in golden set testMatch: /.*\.setup\.ts/ }, { name: 'chromium', use: { ...devices['Desktop Chrome'], // Use prepared auth state. storageState: 'playwright/.auth/user.json', }, dependencies: ['setup'], }, { name: 'firefox', use: { ...devices['Desktop Firefox'], // Use prepared auth state. storageState: 'playwright/.auth/user.json', }, dependencies: ['setup'], }, ], }); ``` Tests start already authenticated because we specified `storageState` in the config. ```js import { test } from '@playwright/test'; test('test', async ({ page }) => { // page is authenticated }); ``` Note that you need to delete the stored state when it expires. If you don't need to keep the state between test runs, write the browser state under testProject.outputDir, which is automatically cleaned up before every test run. auth#basic-shared-account-in-all-tests~2rank 2 · score 0.767not in golden set own. await page.goto('https://github.com/login'); await page.getByLabel('Username or email address').fill('username'); await page.getByLabel('Password').fill('password'); await page.getByRole('button', { name: 'Sign in' }).click(); // Wait until the page receives the cookies. // // Sometimes login flow sets cookies in the process of several redirects. // Wait for the final URL to ensure that the cookies are actually set. await page.waitForURL('https://github.com/'); // Alternatively, you can wait until the page reaches a state where all cookies are set. await expect(page.getByRole('button', { name: 'View profile and more' })).toBeVisible(); // End of authentication steps. await page.context().storageState({ path: authFile }); }); ``` Create a new `setup` project in the config and declare it as a [dependency](./test-projects.md#dependencies) for all your testing projects. This project will always run and authenticate before all the tests. All testing projects should use the authenticated state as `storageState`. ```js import { defineConfig, devices } from '@playwright/test'; export default defineConfig({ projects: [ // Setup project { name: 'setup', testMatch: /.*\.setup\.ts/ }, { name: 'chromium', use: { ...devices['Desktop Chrome'], // Use prepared auth state. storageState: 'playwright/.auth/user.json', }, dependencies: ['setup'], }, { name: 'firefox', use: { ...devices['Desktop Firefox'], // Use test_parallel#parallelize-tests-in-a-single-file~2rank 3 · score 0.742not in golden set export default defineConfig({ // runs all tests in all files of a specific project in parallel projects: [ { name: 'chromium', use: { ...devices['Desktop Chrome'] }, fullyParallel: true, }, ] }); ``` Expected but not retrieved: Score across history 1 flip(s), all across config changes #1 0.99#2 0.83#3 0.50 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass degraded | cfg-run-one-projectHow do I run only the firefox project from the command line? | config | 0.735 | -0.098 | expected_citation_overlap | 0.0 | |||||||||||||||||||||||||||||||||||||||||
Question How do I run only the firefox project from the command line? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Actual answer To run tests on the Firefox project, use the `--project` flag followed by `firefox` [running_tests#run-tests-on-different-browsers]. Checks
Retrieval metrics hit_rate@3 0.000precision@3 0.000recall@3 0.000reciprocal_rank 0.000 Retrieved chunks running_tests#run-tests-on-different-browsersrank 1 · score 0.740not in golden set To specify which browser you would like to run your tests on, use the `--project` flag followed by the browser name. ```bash npx playwright test --project webkit ``` To specify multiple browsers to run your tests on, use the `--project` flag multiple times followed by each browser name. ```bash npx playwright test --project webkit --project firefox ``` browsers#run-tests-on-different-browsers-3rank 2 · score 0.724not in golden set Run tests on a specific browser: Run tests on multiple browsers and make it based on the environment variable `BROWSER`: debug#debug-one-test-on-a-specific-browserrank 3 · score 0.719not in golden set To run one test on a specific browser add the name of the test file and the line number of the test you want to debug as well as the `--project` flag followed by the name of the project. ```bash npx playwright test example.spec.ts:10 --project=webkit --debug ``` Expected but not retrieved: Score across history 1 flip(s), all across config changes #1 0.66#2 0.83#3 0.73 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail regressed | cfg-default-reporter-ciWhich reporter is used by default on CI? | config | 0.681 | -0.152 | expected_citation_overlapgrounding | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question Which reporter is used by default on CI? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: list locally, dot on CI. Two chunks state it; either is a hit. Actual answer The `dot` reporter is used by default on CI [test_cli#options-4, test_reporters#reporters-on-ci]. Attribution mixedretrieval was partial (recall 0.5); failing: expected_citation_overlap, grounding. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 0.500reciprocal_rank 0.333 Retrieved chunks test_cli#options-4rank 1 · score 0.792not in golden set | Option | Description | | :--- | :--- | | `-c, --config <file>` | Configuration file. Can be used to specify additional configuration for the output report | | `--reporter <reporter>` | Reporter to use, comma-separated, can be "list", "line", "dot", "json", "junit", "null", "github", "html", "blob" (default: "list" locally and "dot" on CI) | test_reporters#reporters-on-cirank 2 · score 0.779not in golden set You can use different reporters locally and on CI. For example, using concise `'dot'` reporter avoids too much output. This is the default on CI. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Concise 'dot' for CI, default 'list' when running locally reporter: process.env.CI ? 'dot' : 'list', }); ``` test_reporters#list-reporterrank 3 · score 0.727expected List reporter is default (except on CI where the `dot` reporter is default). It prints a line for each test being run. ```bash npx playwright test --reporter=list ``` ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ reporter: 'list', }); ``` Here is an example output in the middle of a test run. Failures will be listed at the end by default. ```bash npx playwright test --reporter=list Running 124 tests using 6 workers 1 ✓ should access error in env (438ms) 2 ✓ handle long test names (515ms) 3 x 1) render expected (691ms) 4 ✓ should timeout (932ms) 5 should repeat each: 6 ✓ should respect enclosing .gitignore (569ms) 7 should teardown env after timeout: 8 should respect excluded tests: 9 ✓ should handle env beforeEach error (638ms) 10 should respect enclosing .gitignore: ``` You can opt into the step rendering via passing the following config option: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ reporter: [['list', { printSteps: true }]], }); ``` You can print failures inline as soon as they are available instead of Expected but not retrieved: Score across history 1 flip(s), all across config changes #1 0.83#2 0.83#3 0.68 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail improved | cfg-html-report-openHow do I stop the HTML report from opening in a browser automatically after a failed run? | config | 0.897 | +0.069 | grounding | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question How do I stop the HTML report from opening in a browser automatically after a failed run? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: reporter [['html', { open: 'never' }]] or PLAYWRIGHT_HTML_OPEN. Actual answer You can stop the HTML report from opening automatically by setting the `open` property to `'never'` in the Playwright config [test_reporters#html-reporter]. Attribution generationevery expected chunk was retrieved (recall 1.0) yet the case fails on grounding. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 0.333 Retrieved chunks intro#html-test-reportsrank 1 · score 0.819not in golden set After a test run, the [HTML Reporter](./test-reporters.md#html-reporter) provides a dashboard filterable by the browser, passed, failed, skipped, flaky and more. Click a test to inspect errors, attachments and steps. It auto-opens only when failures occur; open manually with the command below. <Tabs groupId="js-package-manager" defaultValue="npm" values={[ {label: 'npm', value: 'npm'}, {label: 'yarn', value: 'yarn'}, {label: 'pnpm', value: 'pnpm'} ] }> <TabItem value="npm"> ```bash npx playwright show-report ``` </TabItem> <TabItem value="yarn"> ```bash yarn playwright show-report ``` </TabItem> <TabItem value="pnpm"> ```bash pnpm exec playwright show-report ``` </TabItem> </Tabs>  running_tests#test-reportsrank 2 · score 0.794not in golden set The [HTML Reporter](./test-reporters.md#html-reporter) shows you a full report of your tests allowing you to filter the report by browsers, passed tests, failed tests, skipped tests, and flaky tests. By default, the HTML report opens automatically if some tests failed, otherwise you can open it with the following command. ```bash npx playwright show-report ```  You can filter and search for tests as well as click on each test to see the test errors and explore each step of the test.  test_reporters#html-reporterrank 3 · score 0.764expected HTML reporter produces a self-contained folder that contains report for the test run that can be served as a web page. ```bash npx playwright test --reporter=html ``` By default, HTML report is opened automatically if some of the tests failed. You can control this behavior via the `open` property in the Playwright config or the `PLAYWRIGHT_HTML_OPEN` environmental variable. The possible values for that property are `always`, `never` and `on-failure` (default). You can also configure `host` and `port` that are used to serve the HTML report. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ reporter: [['html', { open: 'never' }]], }); ``` By default, report is written into the `playwright-report` folder in the current working directory. One can override that location using the `PLAYWRIGHT_HTML_OUTPUT_DIR` environment variable or a reporter configuration. In configuration file, pass options directly: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ reporter: [['html', { outputFolder: 'my-report' }]], }); ``` If you are uploading attachments from a data folder to another location, you can use `attachmentsBaseURL` option to let html report know where to look Score across history 1 flip(s), all across config changes #1 0.83#2 0.83#3 0.90 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass degraded | cfg-github-annotationsIs there a reporter that annotates failures directly in GitHub Actions? | config | 0.986 | -0.011 | - | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question Is there a reporter that annotates failures directly in GitHub Actions? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Actual answer Yes, you can use the built-in `github` reporter to get automatic failure annotations when running in GitHub actions [test_reporters#github-actions-annotations]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 1.000 Retrieved chunks test_reporters#github-actions-annotationsrank 1 · score 0.871expected You can use the built in `github` reporter to get automatic failure annotations when running in GitHub actions. Note that all other reporters work on GitHub Actions as well, but do not provide annotations. Also, it is not recommended to use this annotation type if running your tests with a matrix strategy as the stack trace failures will multiply and obscure the GitHub file view. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // 'github' for GitHub Actions CI to generate annotations, plus a concise 'dot' // default 'list' when running locally reporter: process.env.CI ? 'github' : 'list', }); ``` The `github` reporter accepts `omitTags` (or the `PLAYWRIGHT_GITHUB_OMIT_TAGS` environment variable) to suppress test tags in its annotations, for example `reporter: [['github', { omitTags: true }]]`. ci_intro#opening-the-workflowsrank 2 · score 0.759not in golden set Click on the **Actions** tab to see the workflows. Here you see if your tests have passed or failed. ###### <img src="https://user-images.githubusercontent.com/13063165/183423783-58bf2008-514e-4f96-9c12-c9a55703960c.png" alt="opening the workflow" width="1678" height="1207" /> ###### <img src="https://github.com/microsoft/playwright/assets/13063165/71793c09-0815-4faa-866b-85684a1f87e5" alt="opening the workflow" width="2812" height="824" /> On Pull Requests you can also click on the **Details** link in the [PR status check](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/collaborating-on-repositories-with-code-quality-features/about-status-checks). <img height="404" width="1290" alt="pr status checked" src="https://user-images.githubusercontent.com/13063165/183722462-17a985db-0e10-4205-b16c-8aaac36117b9.png" /> ci_intro#viewing-test-logsrank 3 · score 0.755not in golden set Clicking on the workflow run shows you all the actions that GitHub performed and clicking on **Run Playwright tests** shows the error messages, what was expected and what was received as well as the call log. ###### <img src="https://user-images.githubusercontent.com/13063165/183423783-58bf2008-514e-4f96-9c12-c9a55703960c.png" alt="Viewing Test Logs" width="1678" height="1207" /> ###### <img src="https://github.com/microsoft/playwright/assets/13063165/ba2d8d7b-ffce-42de-95e0-bcb35c421975" alt="viewing the test logs" width="2794" height="1858" /> Score across history no pass/fail flips #1 1.00#2 1.00#3 0.99 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass degraded | cfg-webserverCan Playwright start my app's dev server before the tests instead of me running it separately? | config | 0.844 | -0.071 | - | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question Can Playwright start my app's dev server before the tests instead of me running it separately? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Actual answer Yes, Playwright includes a `webServer` option in its configuration file to launch a local dev server before running your tests [test_webserver#introduction]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.667recall@3 1.000reciprocal_rank 1.000 Retrieved chunks test_webserver#introductionrank 1 · score 0.871expected Playwright comes with a `webServer` option in the config file which gives you the ability to launch a local dev server before running your tests. This is ideal for when writing your tests during development and when you don't have a staging or production url to test against. test_webserver#configuring-a-web-serverrank 2 · score 0.847expected Use the `webServer` property in your Playwright config to launch a development web server during the tests. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Run your local dev server before starting the tests webServer: { command: 'npm run start', url: 'http://localhost:3000', reuseExistingServer: !process.env.CI, stdout: 'ignore', stderr: 'pipe', }, }); ``` | Property | Description | | :- | :- | | testConfig.webServer | Launch a development web server (or multiple) during the tests. | | `command`| Shell command to start the local dev server of your app. | | `cwd` | Current working directory of the spawned process, defaults to the directory of the configuration file. | | `env` | Environment variables for the command. Defaults to inheriting `process.env` with `PLAYWRIGHT_TEST=1` added. | | `gracefulShutdown` | How to shut down the process. If unspecified, the process group is forcefully `SIGKILL`ed. If set to `{ signal: 'SIGTERM', timeout: 500 }`, the process group is sent a `SIGTERM` signal, followed by `SIGKILL` if it doesn't exit within 500ms. You can also use `SIGINT` as the signal instead. A test_webserver#adding-a-server-timeoutrank 3 · score 0.826not in golden set Webservers can sometimes take longer to boot up. In this case, you can increase the timeout to wait for the server to start. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Rest of your config... // Run your local dev server before starting the tests webServer: { command: 'npm run start', url: 'http://localhost:3000', reuseExistingServer: !process.env.CI, timeout: 120 * 1000, }, }); ``` Score across history no pass/fail flips #1 0.92#2 0.91#3 0.84 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass degraded | cfg-update-snapshotsThe UI changed on purpose and my screenshot comparisons fail. How do I regenerate the baselines? | config | 0.944 | -0.056 | - | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question The UI changed on purpose and my screenshot comparisons fail. How do I regenerate the baselines? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Actual answer You can update the reference screenshot by using the `--update-snapshots` flag [test_snapshots#updating-screenshots]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 1.000 Retrieved chunks test_snapshots#updating-screenshotsrank 1 · score 0.749expected Sometimes you need to update the reference screenshot, for example when the page has changed. Do this with the `--update-snapshots` flag. ```bash npx playwright test --update-snapshots ``` test_ui_mode#attachmentsrank 2 · score 0.719not in golden set The "Attachments" tab allows you to explore attachments. If you're doing [visual regression testing](./test-snapshots.md), you'll be able to compare screenshots by examining the image diff, the actual image and the expected image. When you click on the expected image you can use the slider to slide one image over the other so you can easily see the differences in your screenshots. <img src="https://github.com/microsoft/playwright/assets/13063165/bb83b406-84ed-4380-a96c-0e62d1388093" alt="ui mode with attachments" width="3606" height="2228" /> trace_viewer#attachmentsrank 3 · score 0.715not in golden set The "Attachments" tab allows you to explore attachments. If you're doing [visual regression testing](./test-snapshots.md), you'll be able to compare screenshots by examining the image diff, the actual image and the expected image. When you click on the expected image you can use the slider to slide one image over the other so you can easily see the differences in your screenshots. <img src="https://github.com/microsoft/playwright/assets/13063165/4386178a-5808-4fa8-9436-315350a23b04" alt="attachments tab in trace viewer" width="3598" height="2218" /> Score across history no pass/fail flips #1 1.00#2 1.00#3 0.94 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail regressed | cfg-max-diff-pixelsHow do I allow a small number of differing pixels in toHaveScreenshot? | config | 0.852 | -0.146 | grounding | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question How do I allow a small number of differing pixels in toHaveScreenshot? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Actual answer You can allow a small number of differing pixels by using the `maxDiffPixels` option in `toHaveScreenshot` or by specifying it globally in your Playwright config [test_snapshots#maxdiffpixels]. Attribution generationevery expected chunk was retrieved (recall 1.0) yet the case fails on grounding. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 0.333 Retrieved chunks test_snapshots#stylepathrank 1 · score 0.729not in golden set You can apply a custom stylesheet to your page while taking screenshot. This allows filtering out dynamic or volatile elements, hence improving the screenshot determinism. ```css iframe { visibility: hidden; } ``` ```js import { test, expect } from '@playwright/test'; test('example test', async ({ page }) => { await page.goto('https://playwright.dev'); await expect(page).toHaveScreenshot({ stylePath: path.join(__dirname, 'screenshot.css') }); }); ``` If you'd like to share the default value among all the tests in the project, you can specify it in the playwright config, either globally or per project: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ expect: { toHaveScreenshot: { stylePath: './screenshot.css' }, }, }); ``` screenshots#capture-into-bufferrank 2 · score 0.726not in golden set Rather than writing into a file, you can get a buffer with the image and post-process it or pass it to a third party pixel diff facility. ```js const buffer = await page.screenshot(); console.log(buffer.toString('base64')); ``` test_snapshots#maxdiffpixelsrank 3 · score 0.723expected Playwright Test uses the [pixelmatch](https://github.com/mapbox/pixelmatch) library. You can [pass various options](./api/class-pageassertions.md#page-assertions-to-have-screenshot-1) to modify its behavior: ```js import { test, expect } from '@playwright/test'; test('example test', async ({ page }) => { await page.goto('https://playwright.dev'); await expect(page).toHaveScreenshot({ maxDiffPixels: 100 }); }); ``` If you'd like to share the default value among all the tests in the project, you can specify it in the playwright config, either globally or per project: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ expect: { toHaveScreenshot: { maxDiffPixels: 100 }, }, }); ``` Score across history 1 flip(s), all across config changes #1 0.83#2 1.00#3 0.85 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail regressed | cfg-video-retainHow do I record video only for tests that fail? | config | 0.909 | -0.091 | grounding | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question How do I record video only for tests that fail? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Actual answer To record video only for tests that fail, you can use the `'retain-on-failure'` option for the `video` setting in your Playwright configuration [videos#record-video]. Attribution generationevery expected chunk was retrieved (recall 1.0) yet the case fails on grounding. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 1.000 Retrieved chunks videos#record-videorank 1 · score 0.817expected Playwright Test can record videos for your tests, controlled by the `video` option in your Playwright config. By default videos are off. - `'off'` - Do not record video. - `'on'` - Record video for each test. - `'retain-on-failure'` - Record video for each test, but remove all videos from successful test runs. - `'on-first-retry'` - Record video only when retrying a test for the first time. Video files will appear in the test output directory, typically `test-results`. See testOptions.video for advanced video configuration. Videos are saved upon [browser context](./browser-contexts.md) closure at the end of a test. If you create a browser context manually, make sure to await browserContext.close(). ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ use: { video: 'on-first-retry', }, }); ``` ```js const context = await browser.newContext({ recordVideo: { dir: 'videos/' } }); // Make sure to await close, so that videos are saved. await context.close(); ``` You can also specify video size and annotation. The video size defaults to the viewport size scaled down to fit 800x800. The video of the viewport is placed test_use_options#recording-optionsrank 2 · score 0.775not in golden set With Playwright you can capture screenshots, record videos as well as traces of your test. By default these are turned off but you can enable them by setting the `screenshot`, `video` and `trace` options in your `playwright.config.js` file. Trace files, screenshots and videos will appear in the test output directory, typically `test-results`. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ use: { // Capture screenshot after each test failure. screenshot: 'only-on-failure', // Record trace only when retrying a test for the first time. trace: 'on-first-retry', // Record video only when retrying a test for the first time. video: 'on-first-retry' }, }); ``` | Option | Description | | :- | :- | | testOptions.screenshot | Capture [screenshots](./screenshots.md) of your test. Options include `'off'`, `'on'` and `'only-on-failure'` | | testOptions.trace | Playwright can produce test traces while running the tests. Later on, you can view the trace and get detailed information about Playwright execution by opening [Trace Viewer](./trace-viewer.md). Options include: `'off'`, `'on'`, `'retain-on-failure'` and `'on-first-retry'` | | testOptions.video | Playwright can record [videos](./videos.md) for your tests. Options include: `'off'`, trace_viewer#run-trace-only-on-failurerank 3 · score 0.775not in golden set Setup your tests to record a trace only when the test fails: <Tabs groupId="test-runners" defaultValue="mstest" values={[ {label: 'MSTest', value: 'mstest'}, {label: 'NUnit', value: 'nunit'}, {label: 'xUnit', value: 'xunit'}, {label: 'xUnit v3', value: 'xunit-v3'}, ] }> <TabItem value="nunit"> </TabItem> <TabItem value="mstest"> </TabItem> <TabItem value="xunit"> </TabItem> <TabItem value="xunit-v3"> </TabItem> </Tabs> Score across history 1 flip(s), all across config changes #1 1.00#2 1.00#3 0.91 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail degraded | cfg-tag-testsHow do I tag tests as smoke so I can run just those? | config | 0.000 | -0.831 | keyword_presencerefusal_behaviourexpected_citation_overlap | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question How do I tag tests as smoke so I can run just those? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: Tags must start with @, via the details object or the title. Actual answer I do not have enough information in the provided documents to answer that. Attribution generationevery expected chunk was retrieved (recall 1.0) yet the case fails on keyword_presence, refusal_behaviour, expected_citation_overlap. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 0.500 Retrieved chunks test_projects#splitting-tests-into-projectsrank 1 · score 0.774not in golden set We can split tests into projects and use filters to run a subset of tests. For example, we can create a project that runs tests using a filter matching all tests with a specific file name. We can then have another group of tests that ignore specific test files. Here is an example that defines a common timeout and two projects. The "Smoke" project runs a small subset of tests without retries, and "Default" project runs all other tests with retries. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ timeout: 60000, // Timeout is shared between all tests. projects: [ { name: 'Smoke', testMatch: /.*smoke.spec.ts/, retries: 0, }, { name: 'Default', testIgnore: /.*smoke.spec.ts/, retries: 2, }, ], }); ``` test_annotations#tag-testsrank 2 · score 0.767expected Sometimes you want to tag your tests as `@fast` or `@slow`, and then filter by tag in the test report. Or you might want to only run tests that have a certain tag. To tag a test, either provide an additional details object when declaring a test, or add `@`-token to the test title. Note that tags must start with `@` symbol. ```js import { test, expect } from '@playwright/test'; test('test login page', { tag: '@fast', }, async ({ page }) => { // ... }); test('test full report @slow', async ({ page }) => { // ... }); ``` You can also tag all tests in a group or provide multiple tags: ```js import { test, expect } from '@playwright/test'; test.describe('group', { tag: '@report', }, () => { test('test report header', async ({ page }) => { // ... }); test('test full report', { tag: ['@slow', '@vrt'], }, async ({ page }) => { // ... }); }); ``` You can now run tests that have a particular tag with [`--grep`](./test-cli.md#all-options) command line option. ```bash npx playwright test --grep test_sharding#merging-reports-from-multiple-environmentsrank 3 · score 0.760not in golden set If you want to run the same tests in multiple environments, as opposed to shard your tests onto multiple machines, you need to differentiate these environments. In this case, it is useful to specify the testConfig.tag property, to tag all tests with the environment name. This tag will be automatically picked up by the blob report and later on by the merge tool. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ reporter: process.env.CI ? 'blob' : 'html', tag: process.env.CI_ENVIRONMENT_NAME, // for example "@APIv2" }); ``` Score across history no pass/fail flips #1 0.66#2 0.83#3 0.00 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail degraded | cfg-focus-testHow do I run only one test while I'm debugging it? | config | 0.557 | -0.107 | keyword_presenceexpected_citation_overlap | 0.0 | |||||||||||||||||||||||||||||||||||||||||
Question How do I run only one test while I'm debugging it? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Actual answer To debug one test, run the test command followed by the test file name, the line number, and the `--debug` flag [debug#debug-one-test-on-all-browsers]. For example, use `npx playwright test example.spec.ts:10 --debug` [debug#debug-one-test-on-all-browsers]. Attribution retrievalno expected chunk was retrieved (hit rate 0), so the generator never saw the evidence. Checks
Retrieval metrics hit_rate@3 0.000precision@3 0.000recall@3 0.000reciprocal_rank 0.000 Retrieved chunks debug#debug-one-test-on-all-browsersrank 1 · score 0.790not in golden set To debug one test on a specific line, run the test command followed by the name of the test file and the line number of the test you want to debug, followed by the `--debug` flag. This will run a single test in each browser configured in your [`playwright.config`](./test-projects.md#configure-projects-for-multiple-browsers) and open the inspector. ```bash npx playwright test example.spec.ts:10 --debug ``` running_tests#introductionrank 2 · score 0.788not in golden set With Playwright you can run a single test, a set of tests, or all tests. Tests can be run on one browser or multiple browsers using the `--project` flag. Tests run in parallel by default and in headless mode, meaning no browser window opens while running the tests and results appear in the terminal. You can run tests in headed mode using the `--headed` CLI argument, or run your tests in [UI mode](./test-ui-mode.md) using the `--ui` flag to see a full trace of your tests. **You will learn** - [How to run tests from the command line](/running-tests.md#command-line) - [How to debug tests](/running-tests.md#debugging-tests) - [How to open the HTML test reporter](/running-tests.md#test-reports) test_global_setup_teardown#test-filteringrank 3 · score 0.780not in golden set All test filtering options, such as `--grep`/`--grep-invert`, `--shard`, filtering directly by location in the command line, or using [`test.only()`](./api/class-test.md#test-only), directly select the primary tests to be run. If those tests belong to a project with dependencies, all tests from those dependencies will also run. You can pass `--no-deps` command line option to ignore all dependencies and teardowns. Only your directly selected projects will run. Expected but not retrieved: Score across history no pass/fail flips #1 0.66#2 0.66#3 0.56 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail regressed | net-mock-apiHow do I return fake JSON for an API call so the test never hits the real backend? | network | 0.794 | -0.121 | grounding | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question How do I return fake JSON for an API call so the test never hits the real backend? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: page.route() + route.fulfill({ json }). Actual answer To return fake JSON for an API call without hitting the real backend, you can use `page.route` to intercept the call and fulfill it with custom JSON data [mock#mock-api-requests]. For example, call `await route.fulfill({ json })` within the route handler before navigating to the page [mock#mock-api-requests]. Attribution generationevery expected chunk was retrieved (recall 1.0) yet the case fails on grounding. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.667recall@3 1.000reciprocal_rank 1.000 Retrieved chunks mock#mock-api-requestsrank 1 · score 0.815expected The following code will intercept all the calls to `*/**/api/v1/fruits` and will return a custom response instead. No requests to the API will be made. The test goes to the URL that uses the mocked route and asserts that mock data is present on the page. ```js test("mocks a fruit and doesn't call api", async ({ page }) => { // Mock the api call before navigating await page.route('*/**/api/v1/fruits', async route => { const json = [{ name: 'Strawberry', id: 21 }]; await route.fulfill({ json }); }); // Go to the page await page.goto('https://demo.playwright.dev/api-mocking'); // Assert that the Strawberry fruit is visible await expect(page.getByText('Strawberry')).toBeVisible(); }); ``` You can see from the trace of the example test that the API was never called, it was however fulfilled with the mock data. <img src="https://github.com/microsoft/playwright/assets/13063165/3dc14cbf-c100-4efc-ac21-d7b52d698b53" alt="api mocking trace" width="2946" height="1902" /> Read more about [advanced networking](./network.md). mock#modify-api-responsesrank 2 · score 0.795not in golden set Sometimes, it is essential to make an API request, but the response needs to be patched to allow for reproducible testing. In that case, instead of mocking the request, one can perform the request and fulfill it with the modified response. In the example below we intercept the call to the fruit API and add a new fruit called 'Loquat', to the data. We then go to the url and assert that this data is there: ```js test('gets the json from api and adds a new fruit', async ({ page }) => { // Get the response and add to it await page.route('*/**/api/v1/fruits', async route => { const response = await route.fetch(); const json = await response.json(); json.push({ name: 'Loquat', id: 100 }); // Fulfill using the original response, while patching the response body // with the given JSON object. await route.fulfill({ response, json }); }); // Go to the page await page.goto('https://demo.playwright.dev/api-mocking'); // Assert that the new fruit is visible await expect(page.getByText('Loquat', { exact: true })).toBeVisible(); }); ``` In the trace of our test we can see that the network#handle-requestsrank 3 · score 0.763expected ```js await page.route('**/api/fetch_data', route => route.fulfill({ status: 200, body: testData, })); await page.goto('https://example.com'); ``` You can mock API endpoints via handling the network requests in your Playwright script. Score across history 1 flip(s), all across config changes #1 0.92#2 0.92#3 0.79 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass degraded | net-abort-imagesCan I block all image requests to make tests faster? | network | 0.922 | -0.076 | - | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question Can I block all image requests to make tests faster? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Actual answer Yes, you can abort requests using `page.route()` and `route.abort()` to block image requests like png and jpeg [network#abort-requests, network#network-mocking]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 1.000 Retrieved chunks network#abort-requestsrank 1 · score 0.789expected You can abort requests using page.route() and route.abort(). ```js await page.route('**/*.{png,jpg,jpeg}', route => route.abort()); // Abort based on the request type await page.route('**/*', route => { return route.request().resourceType() === 'image' ? route.abort() : route.continue(); }); ``` network#network-mockingrank 2 · score 0.736not in golden set You don't have to configure anything to mock network requests. Just define a custom Route that mocks network for a browser context. ```js import { test, expect } from '@playwright/test'; test.beforeEach(async ({ context }) => { // Block any css requests for each test in this file. await context.route(/.css$/, route => route.abort()); }); test('loads page without css', async ({ page }) => { await page.goto('https://playwright.dev'); // ... test goes here }); ``` Alternatively, you can use page.route() to mock network in a single page. ```js import { test, expect } from '@playwright/test'; test('loads page without images', async ({ page }) => { // Block png and jpeg images. await page.route(/(png|jpeg)$/, route => route.abort()); await page.goto('https://playwright.dev'); // ... test goes here }); ``` debug#run-a-test-from-a-specific-breakpointrank 3 · score 0.712not in golden set To speed up the debugging process you can add a page.pause() method to your test. This way you won't have to step through each action of your test to get to the point where you want to debug. ```js await page.pause(); ``` Once you add a `page.pause()` call, run your tests in debug mode. Clicking the "Resume" button in the Inspector will run the test and only stop on the `page.pause()`. <img height="1736" width="2700" alt="test with page.pause" src="https://user-images.githubusercontent.com/13063165/219473050-122be4c2-31d0-4cbd-aa8b-8588e8b781a6.png" /> Score across history no pass/fail flips #1 0.83#2 1.00#3 0.92 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail degraded | net-glob-double-starIn a page.route pattern, what is the difference between * and **? | network | 0.000 | -0.661 | keyword_presencerefusal_behaviourexpected_citation_overlap | 0.0 | |||||||||||||||||||||||||||||||||||||||||
Question In a page.route pattern, what is the difference between * and **? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: A single * matches any characters except /, ** includes /. Actual answer I do not have enough information in the provided documents to answer that. Attribution retrievalno expected chunk was retrieved (hit rate 0), so the generator never saw the evidence. Checks
Retrieval metrics hit_rate@3 0.000precision@3 0.000recall@3 0.000reciprocal_rank 0.000 Retrieved chunks evaluating#evaluation-argumentrank 1 · score 0.720not in golden set Playwright evaluation methods like page.evaluate() take a single optional argument. This argument can be a mix of Serializable values and JSHandle instances. Handles are automatically converted to the value they represent. ```js // A primitive value. await page.evaluate(num => num, 42); // An array. await page.evaluate(array => array.length, [1, 2, 3]); // An object. await page.evaluate(object => object.foo, { foo: 'bar' }); // A single handle. const button = await page.evaluateHandle('window.button'); await page.evaluate(button => button.textContent, button); // Alternative notation using JSHandle.evaluate. await button.evaluate((button, from) => button.textContent.substring(from), 5); // Object with multiple handles. const button1 = await page.evaluateHandle('window.button1'); const button2 = await page.evaluateHandle('window.button2'); await page.evaluate( o => o.button1.textContent + o.button2.textContent, { button1, button2 }); // Object destructuring works. Note that property names must match // between the destructured object and the argument. // Also note the required parenthesis. await page.evaluate( ({ button1, button2 }) => button1.textContent + button2.textContent, { button1, button2 }); // Array works as well. Arbitrary names can be used for destructuring. // Note the required parenthesis. await page.evaluate( ([b1, b2]) => b1.textContent + b2.textContent, [button1, button2]); // Any pages#pagesrank 2 · score 0.710not in golden set Each BrowserContext can have multiple pages. A Page refers to a single tab or a popup window within a browser context. It should be used to navigate to URLs and interact with the page content. ```js // Create a page. const page = await context.newPage(); // Navigate explicitly, similar to entering a URL in the browser. await page.goto('http://example.com'); // Fill an input. await page.locator('#search').fill('query'); // Navigate implicitly by clicking a link. await page.locator('#submit').click(); // Expect a new url. console.log(page.url()); ``` other_locators#xpath-locatorrank 3 · score 0.707not in golden set We recommend prioritizing [user-visible locators](./locators.md#quick-guide) like text or accessible role instead of using XPath that is tied to the implementation and easily break when the page changes. XPath locators are equivalent to calling [`Document.evaluate`](https://developer.mozilla.org/en/docs/Web/API/Document/evaluate). ```js await page.locator('xpath=//button').click(); ``` Any selector string starting with `//` or `..` are assumed to be an xpath selector. For example, Playwright converts `'//html/body'` to `'xpath=//html/body'`. XPath does not pierce shadow roots. Expected but not retrieved: Score across history no pass/fail flips #1 0.67#2 0.66#3 0.00 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass improved | net-redirect-handlerIf a request redirects, does my route handler get called again for the redirected request? | network | 0.810 | +0.020 | exact_match | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question If a request redirects, does my route handler get called again for the redirected request? Expected No. The handler is called once for the original request and the browser follows the redirect on its own. required: Actual answer No, the handler is only called once for the original request, and the browser follows the redirect on its own [network#redirects]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 1.000 Retrieved chunks network#redirectsrank 1 · score 0.809expected Playwright treats a request and its redirects as a single unit. The handler is called once, for the original request, and the browser follows the redirect on its own. response.request() returns the last request in the chain, and request.redirectedFrom() walks it back to the one you intercepted. Headers passed to route.continue() apply to every hop of the chain, except `cookie`, which always comes from the cookie store. Fulfilling with a `3xx` status does not give you a second chance to intercept. Chromium and Firefox follow the redirect without calling your handler, and WebKit rejects the call. To serve different content, fulfill with that content. To send the request elsewhere, pass `url` to route.continue() or route.fallback(). network#how-request-interception-worksrank 2 · score 0.767not in golden set Routes sit between the page and the browser's network stack. The handler runs before the network stack has processed the request: route.continue() passes it on, route.fulfill() answers it without touching the network, and route.abort() fails it. network#network-eventsrank 3 · score 0.727not in golden set You can monitor all the Requests and Responses: ```js // Subscribe to 'request' and 'response' events. page.on('request', request => console.log('>>', request.method(), request.url())); page.on('response', response => console.log('<<', response.status(), response.url())); await page.goto('https://example.com'); ``` Or wait for a network response after the button click with page.waitForResponse(): ```js // Use a glob URL pattern. Note no await. const responsePromise = page.waitForResponse('**/api/fetch_data'); await page.getByText('Update').click(); const response = await responsePromise; ``` Score across history no pass/fail flips #1 0.79#2 0.79#3 0.81 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass degraded | net-msw-missing-eventsI use Mock Service Worker in the app and page.route never sees the requests. What is going on? | network | 0.967 | -0.033 | - | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question I use Mock Service Worker in the app and page.route never sees the requests. What is going on? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: MSW adds its own service worker; set serviceWorkers to 'block'. Actual answer Mock Service Worker adds its own Service Worker that takes over network requests, making them invisible to Playwright's `page.route()` and `browserContext.route()` [network#missing-network-events-and-service-workers]. To fix this, you can disable Service Workers by setting `browser.newContext.serviceWorkers` to `'block'` [network#missing-network-events-and-service-workers]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 1.000 Retrieved chunks network#missing-network-events-and-service-workersrank 1 · score 0.805expected Playwright's built-in browserContext.route() and page.route() allow your tests to natively route requests and perform mocking and interception. If you're using Playwright's native browserContext.route() and page.route(), and it appears network events are missing, disable Service Workers by setting browser.newContext.serviceWorkers to `'block'`. It might be that you are using a mock tool such as Mock Service Worker (MSW). While this tool works out of the box for mocking responses, it adds its own Service Worker that takes over the network requests, hence making them invisible to browserContext.route() and page.route(). If you are interested in both network testing and mocking, consider using built-in browserContext.route() and page.route() for [response mocking](#handle-requests). ###### If you're interested in not solely using Service Workers for testing and network mocking, but in routing and listening for requests made by Service Workers themselves, please see [this guide](./service-workers.md). mock#mock-api-requestsrank 2 · score 0.788not in golden set The following code will intercept all the calls to `*/**/api/v1/fruits` and will return a custom response instead. No requests to the API will be made. The test goes to the URL that uses the mocked route and asserts that mock data is present on the page. ```js test("mocks a fruit and doesn't call api", async ({ page }) => { // Mock the api call before navigating await page.route('*/**/api/v1/fruits', async route => { const json = [{ name: 'Strawberry', id: 21 }]; await route.fulfill({ json }); }); // Go to the page await page.goto('https://demo.playwright.dev/api-mocking'); // Assert that the Strawberry fruit is visible await expect(page.getByText('Strawberry')).toBeVisible(); }); ``` You can see from the trace of the example test that the API was never called, it was however fulfilled with the mock data. <img src="https://github.com/microsoft/playwright/assets/13063165/3dc14cbf-c100-4efc-ac21-d7b52d698b53" alt="api mocking trace" width="2946" height="1902" /> Read more about [advanced networking](./network.md). network#handle-requestsrank 3 · score 0.774not in golden set ```js await page.route('**/api/fetch_data', route => route.fulfill({ status: 200, body: testData, })); await page.goto('https://example.com'); ``` You can mock API endpoints via handling the network requests in your Playwright script. Score across history no pass/fail flips #1 1.00#2 1.00#3 0.97 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail regressed | net-wait-for-responseHow do I wait for a specific API response before asserting on the page? | network | 0.855 | -0.062 | grounding | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question How do I wait for a specific API response before asserting on the page? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Actual answer You can wait for a specific network response by using `page.waitForResponse()` with a glob URL pattern, regular expression, or predicate before performing an action like clicking a button [network#variations, network#network-events]. Then, await the resulting response promise before making assertions [network#variations, network#network-events]. Attribution generationevery expected chunk was retrieved (recall 1.0) yet the case fails on grounding. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.667recall@3 1.000reciprocal_rank 1.000 Retrieved chunks network#variationsrank 1 · score 0.812expected Wait for Responses with page.waitForResponse() ```js // Use a RegExp. Note no await. const responsePromise = page.waitForResponse(/\.jpeg$/); await page.getByText('Update').click(); const response = await responsePromise; // Use a predicate taking a Response object. Note no await. const responsePromise = page.waitForResponse(response => response.url().includes(token)); await page.getByText('Update').click(); const response = await responsePromise; ``` network#network-eventsrank 2 · score 0.811expected You can monitor all the Requests and Responses: ```js // Subscribe to 'request' and 'response' events. page.on('request', request => console.log('>>', request.method(), request.url())); page.on('response', response => console.log('<<', response.status(), response.url())); await page.goto('https://example.com'); ``` Or wait for a network response after the button click with page.waitForResponse(): ```js // Use a glob URL pattern. Note no await. const responsePromise = page.waitForResponse('**/api/fetch_data'); await page.getByText('Update').click(); const response = await responsePromise; ``` mock_browser#creating-mocksrank 3 · score 0.805not in golden set Since the page may be calling the API very early while loading it's important to setup all the mocks before the page started loading. The easiest way to achieve that is to call page.addInitScript(): ```js await page.addInitScript(() => { const mockBattery = { level: 0.75, charging: true, chargingTime: 1800, dischargingTime: Infinity, addEventListener: () => { } }; // Override the method to always return mock battery info. window.navigator.getBattery = async () => mockBattery; }); ``` Once this is done you can navigate the page and check its UI state: ```js // Configure mock API before each test. test.beforeEach(async ({ page }) => { await page.addInitScript(() => { const mockBattery = { level: 0.90, charging: true, chargingTime: 1800, // seconds dischargingTime: Infinity, addEventListener: () => { } }; // Override the method to always return mock battery info. window.navigator.getBattery = async () => mockBattery; }); }); test('show battery status', async ({ page }) => { await page.goto('/'); await expect(page.locator('.battery-percentage')).toHaveText('90%'); await expect(page.locator('.battery-status')).toHaveText('Adapter'); await expect(page.locator('.battery-fully')).toHaveText('00:30'); }); ``` Score across history 1 flip(s), all across config changes #1 0.92#2 0.92#3 0.85 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass improved | net-offline-webrtcDoes the offline option also cut WebRTC connections? | network | 0.782 | +0.021 | exact_matchfuzzy_match | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question Does the offline option also cut WebRTC connections? Expected No. Offline emulation affects the regular network stack only; established RTCPeerConnections keep flowing. required: Actual answer No, offline emulation does not affect WebRTC traffic, as established `RTCPeerConnection`s will keep sending and receiving media over UDP [emulation#offline]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 1.000 Retrieved chunks emulation#offlinerank 1 · score 0.817expected Emulate the network being offline. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ use: { offline: true }, }); ``` Offline emulation only affects requests that go through the browser's regular network stack, such as page navigations, `fetch()`, `XMLHttpRequest` and WebSockets. It does not affect WebRTC traffic: established `RTCPeerConnection`s keep sending and receiving media over UDP. To test WebRTC connection loss, interrupt the connection outside the browser, for example by stopping the TURN server or using an OS-level firewall. test_use_options#network-optionsrank 2 · score 0.671not in golden set Available options to configure networking: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ use: { // Whether to automatically download all the attachments. acceptDownloads: false, // An object containing additional HTTP headers to be sent with every request. extraHTTPHeaders: { 'X-My-Header': 'value', }, // Credentials for HTTP authentication. httpCredentials: { username: 'user', password: 'pass', }, // Whether to ignore HTTPS errors during navigation. ignoreHTTPSErrors: true, // Whether to emulate network being offline. offline: true, // Proxy settings used for all pages in the test. proxy: { server: 'http://myproxy.com:3128', bypass: 'localhost', }, }, }); ``` | Option | Description | | :- | :- | | testOptions.acceptDownloads | Whether to automatically download all the attachments, defaults to `true`. [Learn more](./downloads.md) about working with downloads. | | testOptions.extraHTTPHeaders | An object containing additional HTTP headers to be sent with every request. All header values must be strings. | | testOptions.httpCredentials | Credentials for [HTTP authentication](./network.md#http-authentication). | | testOptions.ignoreHTTPSErrors | Whether to ignore HTTPS errors during navigation. | | testOptions.offline | Whether to emulate network being offline. | | testOptions.proxy | mock#mock-websocketsrank 3 · score 0.664not in golden set The following code will intercept WebSocket connections and mock entire communication over the WebSocket, instead of connecting to the server. This example responds to a `"request"` with a `"response"`. ```js await page.routeWebSocket('wss://example.com/ws', ws => { ws.onMessage(message => { if (message === 'request') ws.send('response'); }); }); ``` Alternatively, you may want to connect to the actual server, but intercept messages in-between and modify or block them. Here is an example that modifies some of the messages sent by the page to the server, and leaves the rest unmodified. ```js await page.routeWebSocket('wss://example.com/ws', ws => { const server = ws.connectToServer(); ws.onMessage(message => { if (message === 'request') server.send('request2'); else server.send(message); }); }); ``` For more details, see WebSocketRoute. Score across history no pass/fail flips #1 0.79#2 0.76#3 0.78 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass degraded | net-download-eventHow do I get hold of a file the page downloads during a test? | network | 0.928 | -0.069 | - | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question How do I get hold of a file the page downloads during a test? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: waitForEvent('download') then download.saveAs(). Actual answer You can obtain a downloaded file by waiting for the `download` event using `page.waitForEvent('download')` and then saving it with `download.saveAs()` [downloads#introduction]. Alternatively, you can handle the event with `page.on('download', ...)` and use `download.path()` [downloads#variations]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 1.000 Retrieved chunks downloads#introductionrank 1 · score 0.799expected For every attachment downloaded by the page, the 'download' event event is emitted. All these attachments are downloaded into a temporary folder. You can obtain the download url, file name and payload stream using the Download object from the event. You can specify where to persist downloaded files using the browserType.launch.downloadsPath option in browserType.launch(). Downloaded files are deleted when the browser context that produced them is closed. Here is the simplest way to handle the file download: ```js // Start waiting for download before clicking. Note no await. const downloadPromise = page.waitForEvent('download'); await page.getByText('Download file').click(); const download = await downloadPromise; // Wait for the download process to complete and save the downloaded file somewhere. await download.saveAs('/path/to/save/at/' + download.suggestedFilename()); ``` downloads#variationsrank 2 · score 0.760not in golden set If you have no idea what initiates the download, you can still handle the event: ```js page.on('download', download => download.path().then(console.log)); ``` Note that handling the event forks the control flow and makes the script harder to follow. Your scenario might end while you are downloading a file since your main control flow is not awaiting for this operation to resolve. For uploading files, see the [uploading files](./input.md#upload-files) section. aria_snapshots#snapshots-as-separate-filesrank 3 · score 0.739not in golden set To store your snapshots in a separate file, use the `toMatchAriaSnapshot` method with the `name` option, specifying a `.aria.yml` file extension. ```js await expect(page.getByRole('main')).toMatchAriaSnapshot({ name: 'main.aria.yml' }); ``` By default, snapshots from a test file `example.spec.ts` are placed in the `example.spec.ts-snapshots` directory. As snapshots should be the same across browsers, only one snapshot is saved even if testing with multiple browsers. Should you wish, you can customize the [snapshot path template](./api/class-testconfig#test-config-snapshot-path-template) using the following configuration: ```js export default defineConfig({ expect: { toMatchAriaSnapshot: { pathTemplate: '__snapshots__/{testFilePath}/{arg}{ext}', }, }, }); ``` Score across history 1 flip(s), all across config changes #1 0.67#2 1.00#3 0.93 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail regressed | net-new-tabA link opens in a new tab. How do I get a handle on that new page? | network | 0.912 | -0.085 | grounding | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question A link opens in a new tab. How do I get a handle on that new page? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: context.waitForEvent('page'). Actual answer You can use `context.waitForEvent('page')` to wait for and get the new page that is created in the context [pages#handling-new-pages]. Alternatively, you can listen for new pages using `context.on('page', ...)` [pages#handling-new-pages]. Attribution generationevery expected chunk was retrieved (recall 1.0) yet the case fails on grounding. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 1.000 Retrieved chunks pages#handling-new-pagesrank 1 · score 0.791expected The `page` event on browser contexts can be used to get new pages that are created in the context. This can be used to handle new pages opened by `target="_blank"` links. ```js // Start waiting for new page before clicking. Note no await. const pagePromise = context.waitForEvent('page'); await page.getByText('open new tab').click(); const newPage = await pagePromise; // Interact with the new page normally. await newPage.getByRole('button').click(); console.log(await newPage.title()); ``` If the action that triggers the new page is unknown, the following pattern can be used. ```js // Get all new pages (including popups) in the context context.on('page', async page => { await page.waitForLoadState(); console.log(await page.title()); }); ``` pages#pagesrank 2 · score 0.726not in golden set Each BrowserContext can have multiple pages. A Page refers to a single tab or a popup window within a browser context. It should be used to navigate to URLs and interact with the page content. ```js // Create a page. const page = await context.newPage(); // Navigate explicitly, similar to entering a URL in the browser. await page.goto('http://example.com'); // Fill an input. await page.locator('#search').fill('query'); // Navigate implicitly by clicking a link. await page.locator('#submit').click(); // Expect a new url. console.log(page.url()); ``` pages#handling-popupsrank 3 · score 0.705not in golden set If the page opens a pop-up (e.g. pages opened by `target="_blank"` links), you can get a reference to it by listening to the `popup` event on the page. This event is emitted in addition to the `browserContext.on('page')` event, but only for popups relevant to this page. ```js // Start waiting for popup before clicking. Note no await. const popupPromise = page.waitForEvent('popup'); await page.getByText('open the popup').click(); const popup = await popupPromise; // Interact with the new popup normally. await popup.getByRole('button').click(); console.log(await popup.title()); ``` If the action that triggers the popup is unknown, the following pattern can be used. ```js // Get all popups when they open page.on('popup', async popup => { await popup.waitForLoadState(); console.log(await popup.title()); }); ``` Score across history 1 flip(s), all across config changes #1 1.00#2 1.00#3 0.91 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail degraded | net-dialog-defaultWhat happens to an alert() dialog if my test never handles it? | network | 0.741 | -0.089 | keyword_presence | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question What happens to an alert() dialog if my test never handles it? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Actual answer By default, dialogs like `alert()` are automatically dismissed by Playwright if there is no listener for the 'dialog' event [dialogs#alert-confirm-prompt-dialogs]. Attribution generationevery expected chunk was retrieved (recall 1.0) yet the case fails on keyword_presence. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 0.500 Retrieved chunks test_assertions#auto-retrying-assertionsrank 1 · score 0.734not in golden set The following assertions will retry until the assertion passes, or the assertion timeout is reached. Note that retrying assertions are async, so you must `await` them. | Assertion | Description | | :- | :- | | [await expect(locator).toBeAttached()](./api/class-locatorassertions.md#locator-assertions-to-be-attached) | Element is attached | | [await expect(locator).toBeChecked()](./api/class-locatorassertions.md#locator-assertions-to-be-checked) | Checkbox is checked | | [await expect(locator).toBeDisabled()](./api/class-locatorassertions.md#locator-assertions-to-be-disabled) | Element is disabled | | [await expect(locator).toBeEditable()](./api/class-locatorassertions.md#locator-assertions-to-be-editable) | Element is editable | | [await expect(locator).toBeEmpty()](./api/class-locatorassertions.md#locator-assertions-to-be-empty) | Container is empty | | [await expect(locator).toBeEnabled()](./api/class-locatorassertions.md#locator-assertions-to-be-enabled) | Element is enabled | | [await expect(locator).toBeFocused()](./api/class-locatorassertions.md#locator-assertions-to-be-focused) | Element is focused | | [await expect(locator).toBeHidden()](./api/class-locatorassertions.md#locator-assertions-to-be-hidden) | Element is not visible | | [await expect(locator).toBeInViewport()](./api/class-locatorassertions.md#locator-assertions-to-be-in-viewport) | Element intersects viewport | | [await expect(locator).toBeVisible()](./api/class-locatorassertions.md#locator-assertions-to-be-visible) | Element is visible | | [await expect(locator).toContainText()](./api/class-locatorassertions.md#locator-assertions-to-contain-text) | Element contains text | | [await expect(locator).toContainClass()](./api/class-locatorassertions.md#locator-assertions-to-contain-class) | Element has specified CSS classes | | [await expect(locator).toHaveAccessibleDescription()](./api/class-locatorassertions.md#locator-assertions-to-have-accessible-description) | Element has a matching [accessible description](https://w3c.github.io/accname/#dfn-accessible-description) | | [await expect(locator).toHaveAccessibleName()](./api/class-locatorassertions.md#locator-assertions-to-have-accessible-name) | Element has a matching [accessible name](https://w3c.github.io/accname/#dfn-accessible-name) | | [await expect(locator).toHaveAttribute()](./api/class-locatorassertions.md#locator-assertions-to-have-attribute) | Element has a DOM attribute | | [await expect(locator).toHaveClass()](./api/class-locatorassertions.md#locator-assertions-to-have-class) | Element has specified CSS class property | | [await dialogs#alert-confirm-prompt-dialogsrank 2 · score 0.727expected By default, dialogs are auto-dismissed by Playwright, so you don't have to handle them. However, you can register a dialog handler before the action that triggers the dialog to either dialog.accept() or dialog.dismiss() it. ```js page.on('dialog', dialog => dialog.accept()); await page.getByRole('button').click(); ``` the 'dialog' event listener **must handle** the dialog. Otherwise your action will stall, be it locator.click() or something else. That's because dialogs in Web are modals and therefore block further page execution until they are handled. As a result, the following snippet will never resolve: WRONG! ```js page.on('dialog', dialog => console.log(dialog.message())); await page.getByRole('button').click(); // Will hang here ``` If there is no listener for the 'dialog' event, all dialogs are automatically dismissed. dialogs#print-dialogsrank 3 · score 0.727not in golden set In order to assert that a print dialog via [`window.print`](https://developer.mozilla.org/en-US/docs/Web/API/Window/print) was triggered, you can use the following snippet: ```js await page.goto('<url>'); await page.evaluate('(() => {window.waitForPrintDialog = new Promise(f => window.print = f);})()'); await page.getByText('Print it!').click(); await page.waitForFunction('window.waitForPrintDialog'); ``` This will wait for the print dialog to be opened after the button is clicked. Make sure to evaluate the script before clicking the button / after the page is loaded. Score across history no pass/fail flips #1 0.83#2 0.83#3 0.74 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail regressed | tool-trace-on-ciWhat trace setting do the docs recommend for CI? | tooling | 0.706 | -0.119 | expected_citation_overlapgrounding | 0.0 | |||||||||||||||||||||||||||||||||||||||||
Question What trace setting do the docs recommend for CI? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Actual answer The documentation recommends only recording traces for failing tests [ci_intro#viewing-the-trace-3], specifically using the `on-first-retry` setting [trace_viewer_intro#recording-a-trace]. Attribution retrievalno expected chunk was retrieved (hit rate 0), so the generator never saw the evidence. Checks
Retrieval metrics hit_rate@3 0.000precision@3 0.000recall@3 0.000reciprocal_rank 0.000 Retrieved chunks trace_viewer_intro#recording-a-tracerank 1 · score 0.782not in golden set By default the [playwright.config](./trace-viewer.md#tracing-on-ci) file contains the configuration needed to create a `trace.zip` file for each test. Traces are setup to run `on-first-retry`, meaning they run on the first retry of a failed test. Also `retries` are set to 2 when running on CI and 0 locally. This means the traces are recorded on the first retry of a failed test but not on the first run and not on the second retry. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ retries: process.env.CI ? 2 : 0, // set to 2 when running on CI // ... use: { trace: 'on-first-retry', // record traces on first retry of each test }, }); ``` To learn more about available options to record a trace check out our detailed guide on [Trace Viewer](/trace-viewer.md). Traces are normally run in a Continuous Integration (CI) environment, because locally you can use [UI Mode](/test-ui-mode.md) for developing and debugging tests. However, if you want to run traces locally without using [UI Mode](/test-ui-mode.md), you can force tracing to be on with `--trace on`. ```bash npx playwright trace_viewer_intro#what-s-nextrank 2 · score 0.777not in golden set - [Run tests on CI with GitHub Actions](/ci-intro.md) - [Learn more about Trace Viewer](/trace-viewer.md) ci_intro#viewing-the-trace-3rank 3 · score 0.772not in golden set You can upload Traces which get created on your CI like GitHub Actions as artifacts. This requires [starting and stopping the trace](./trace-viewer-intro#recording-a-trace). We recommend only recording traces for failing tests. Once your traces have been uploaded to CI, they can then be downloaded and opened using [trace.playwright.dev](https://trace.playwright.dev), which is a statically hosted variant of the Trace Viewer. You can upload trace files using drag and drop. ######  Expected but not retrieved: Score across history 2 flip(s), all across config changes #1 0.67#2 0.82#3 0.71 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail regressed | tool-open-traceHow do I open a trace.zip file that CI produced? | tooling | 0.705 | -0.292 | keyword_presencegrounding | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question How do I open a trace.zip file that CI produced? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Actual answer You can download the trace file produced on CI and open it by uploading it via drag and drop at [trace.playwright.dev](https://trace.playwright.dev) [ci_intro#viewing-the-trace-3]. Alternatively, you can open a saved trace using the Playwright CLI or directly in the browser at the same website [trace_viewer#opening-trace-viewer]. Attribution generationevery expected chunk was retrieved (recall 1.0) yet the case fails on keyword_presence, grounding. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 1.000 Retrieved chunks trace_viewer#opening-trace-viewerrank 1 · score 0.803expected You can open a saved trace using either the Playwright CLI or in the browser at [trace.playwright.dev](https://trace.playwright.dev). Make sure to add the full path to where your `trace.zip` file is located. ```bash npx playwright show-trace path/to/trace.zip ``` ```bash mvn exec:java -e -D exec.mainClass=com.microsoft.playwright.CLI -D exec.args="show-trace trace.zip" ``` ```bash playwright show-trace trace.zip ``` ```bash pwsh bin/Debug/netX/playwright.ps1 show-trace trace.zip ``` ci_intro#viewing-the-trace-3rank 2 · score 0.797not in golden set You can upload Traces which get created on your CI like GitHub Actions as artifacts. This requires [starting and stopping the trace](./trace-viewer-intro#recording-a-trace). We recommend only recording traces for failing tests. Once your traces have been uploaded to CI, they can then be downloaded and opened using [trace.playwright.dev](https://trace.playwright.dev), which is a statically hosted variant of the Trace Viewer. You can upload trace files using drag and drop. ######  trace_viewer#recording-a-trace-3rank 3 · score 0.795not in golden set Traces can be recorded using the browserContext.tracing API as follows: This will record the trace and place it into the file named `trace.zip`. Score across history 1 flip(s), all across config changes #1 1.00#2 1.00#3 0.70 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass degraded | tool-ui-modeWhat is the command to open UI mode? | tooling | 0.961 | -0.036 | - | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question What is the command to open UI mode? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Actual answer To open UI mode, run the command `npx playwright test --ui` in your terminal [test_ui_mode#opening-ui-mode]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 1.000 Retrieved chunks test_ui_mode#opening-ui-moderank 1 · score 0.874expected To open UI mode, run the following command in your terminal: ```bash npx playwright test --ui ``` test_ui_mode#docker-github-codespacesrank 2 · score 0.763not in golden set For Docker and GitHub Codespaces environments, you can run UI mode in the browser. In order for an endpoint to be accessible outside of the container, it needs to be bound to the `0.0.0.0` interface: ```bash npx playwright test --ui-host=0.0.0.0 ``` In the case of GitHub Codespaces, the port gets [forwarded automatically](https://docs.github.com/en/codespaces/developing-in-codespaces/forwarding-ports-in-your-codespace#about-forwarded-ports), so you can open UI mode in the browser by clicking on the link in the terminal. To have a static port, you can pass the `--ui-port` flag: ```bash npx playwright test --ui-port=8080 --ui-host=0.0.0.0 ``` Be aware that when specifying the `--ui-host=0.0.0.0` flag, UI Mode with your traces, the passwords and secrets are accessible from other machines inside your network. In the case of GitHub Codespaces, the ports are only accessible from your account by default. test_ui_mode#running-your-testsrank 3 · score 0.739not in golden set Once you launch UI Mode you will see a list of all your test files. You can run all your tests by clicking the triangle icon in the sidebar. You can also run a single test file, a block of tests or a single test by hovering over the name and clicking on the triangle next to it. <img src="https://github.com/microsoft/playwright/assets/13063165/6b87712f-64a5-4d73-a91d-6562b864712c" alt="running tests in ui mode" width="3960" height="2326" /> Score across history no pass/fail flips #1 1.00#2 1.00#3 0.96 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass degraded | tool-codegenHow do I record a test by clicking through the site instead of writing it by hand? | tooling | 0.747 | -0.086 | expected_citation_overlap | 0.0 | |||||||||||||||||||||||||||||||||||||||||
Question How do I record a test by clicking through the site instead of writing it by hand? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Actual answer To record a new test, click on the **Record new** button from the Testing sidebar, which creates a file and opens a browser window [codegen#record-a-new-test]. Then, go to the URL you wish to test and click around to record your user actions, which will be generated directly as test code in VS Code [codegen#record-a-new-test]. Checks
Retrieval metrics hit_rate@3 0.000precision@3 0.000recall@3 0.000reciprocal_rank 0.000 Retrieved chunks codegen#record-a-new-testrank 1 · score 0.771not in golden set To record a test click on the **Record new** button from the Testing sidebar. This will create a `test-1.spec.ts` file as well as open up a browser window. <img height="1940" width="2770" alt="record new in vs code" src="https://user-images.githubusercontent.com/13063165/220961665-615d0ab8-3f0b-439c-ad0b-0424d9aa154b.png" /> In the browser go to the URL you wish to test and start clicking around to record your user actions. <img src="https://github.com/microsoft/playwright/assets/13063165/1d4c8f37-8325-4816-a665-d0e95e63f509" alt="generating user actions" width="2788" height="1824" /> Playwright will record your actions and generate the test code directly in VS Code. You can also generate assertions by choosing one of the icons in the toolbar and then clicking on an element on the page to assert against. The following assertions can be generated: * `'assert visibility'` to assert that an element is visible * `'assert text'` to assert that an element contains specific text * `'assert value'` to assert that an element has a specific value <img src="https://github.com/microsoft/playwright/assets/13063165/d131eb35-b2ca-4bf4-a8ac-88b6e40dcf07" alt="generating assertions" width="2788" height="1400" /> Once you are done recording click the **cancel** button or close the browser window. You can then inspect your `test-1.spec.ts` file and manually improve it if needed. <img codegen#record-at-cursorrank 2 · score 0.760not in golden set To record from a specific point in your test move your cursor to where you want to record more actions and then click the **Record at cursor** button from the Testing sidebar. If your browser window is not already open then first run the test with 'Show browser' checked and then click the **Record at cursor** button. <img src="https://github.com/microsoft/playwright/assets/13063165/77948ab8-92a2-435f-9833-0944da5ae664" alt="record at cursor in vs code" width="3126" height="1756" /> In the browser window start performing the actions you want to record. <img height="1824" width="2788" alt="add feed the dog to todo app" src="https://user-images.githubusercontent.com/13063165/220960770-6435cec7-1723-42a8-8c1f-8244e2d800c7.png" /> In the test file in VS Code you will see your new generated actions added to your test at the cursor position. <img src="https://github.com/microsoft/playwright/assets/13063165/4f4bb34e-9cda-41fe-bf65-8d8016d84c7f" alt="code from a generated test" width="3180" height="1944" /> trace_viewer#run-trace-only-on-failurerank 3 · score 0.750not in golden set Setup your tests to record a trace only when the test fails: <Tabs groupId="test-runners" defaultValue="mstest" values={[ {label: 'MSTest', value: 'mstest'}, {label: 'NUnit', value: 'nunit'}, {label: 'xUnit', value: 'xunit'}, {label: 'xUnit v3', value: 'xunit-v3'}, ] }> <TabItem value="nunit"> </TabItem> <TabItem value="mstest"> </TabItem> <TabItem value="xunit"> </TabItem> <TabItem value="xunit-v3"> </TabItem> </Tabs> Expected but not retrieved: Score across history no pass/fail flips #1 0.83#2 0.83#3 0.75 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail degraded | tool-install-one-browserHow do I install only WebKit instead of all browsers? | tooling | 0.000 | -0.663 | keyword_presencerefusal_behaviourexpected_citation_overlap | 0.0 | |||||||||||||||||||||||||||||||||||||||||
Question How do I install only WebKit instead of all browsers? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Actual answer I do not have enough information in the provided documents to answer that. Attribution retrievalno expected chunk was retrieved (hit rate 0), so the generator never saw the evidence. Checks
Retrieval metrics hit_rate@3 0.000precision@3 0.000recall@3 0.000reciprocal_rank 0.000 Retrieved chunks test_cli#install-optionsrank 1 · score 0.737not in golden set | Option | Description | | :--- | :--- | | `--force` | Force reinstall of stable browser channels | | `--with-deps` | Install browser system dependencies | | `--dry-run` | Don't perform installation, just print information | | `--only-shell` | Only install chromium-headless-shell instead of full Chromium | | `--no-shell` | Don't install chromium-headless-shell | | `--no-remove` | Don't remove unused browsers | docker#alpinerank 2 · score 0.719not in golden set Browser builds for Firefox and WebKit are built for the [glibc](https://en.wikipedia.org/wiki/Glibc) library. Alpine Linux and other distributions that are based on the [musl](https://en.wikipedia.org/wiki/Musl) standard library are not supported. test_cli#options-3rank 3 · score 0.714not in golden set | Option | Description | | :--- | :--- | | `-b, --browser <name>` | Browser to use: chromium, firefox, or webkit (default: chromium) | | `-h, --host <host>` | Host to serve trace on | | `-p, --port <port>` | Port to serve trace on | Expected but not retrieved: Score across history 1 flip(s), all across config changes #1 1.00#2 0.66#3 0.00 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass improved | tool-docker-imageDoes the official Docker image include the Playwright npm package itself? | tooling | 0.754 | +0.232 | exact_matchfuzzy_match | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question Does the official Docker image include the Playwright npm package itself? Expected No. The image includes the browsers and system dependencies; the Playwright package must be installed separately. required: Actual answer No, the Playwright package/dependency is not included in the Docker image and should be installed separately [docker#introduction]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 0.500 Retrieved chunks docker#image-tagsrank 1 · score 0.856not in golden set See [all available image tags]. We currently publish images with the following tags: - `:v%%VERSION%%` - Playwright v%%VERSION%% release docker image based on Ubuntu 24.04 LTS (Noble Numbat). - `:v%%VERSION%%-noble` - Playwright v%%VERSION%% release docker image based on Ubuntu 24.04 LTS (Noble Numbat). - `:v%%VERSION%%-jammy` - Playwright v%%VERSION%% release docker image based on Ubuntu 22.04 LTS (Jammy Jellyfish). - `:v%%VERSION%%-resolute` - Playwright v%%VERSION%% release docker image based on Ubuntu 26.04 LTS (Resolute Raccoon). It is recommended to always pin your Docker image to a specific version if possible. If the Playwright version in your Docker image does not match the version in your project/tests, Playwright will be unable to locate browser executables. docker#introductionrank 2 · score 0.843expected [Dockerfile.noble] can be used to run Playwright scripts in Docker environment. This image includes the [Playwright browsers](./browsers.md#install-browsers) and [browser system dependencies](./browsers.md#install-system-dependencies). The Playwright package/dependency is not included in the image and should be installed separately. docker#build-your-own-imagerank 3 · score 0.831not in golden set To run Playwright inside Docker, you need to have Node.js, [Playwright browsers](./browsers.md#install-browsers) and [browser system dependencies](./browsers.md#install-system-dependencies) installed. See the following Dockerfile: Score across history 2 flip(s), all across config changes #1 0.80#2 0.52#3 0.75 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass degraded | auth-shared-accountWhat is the recommended way to log in once and reuse the session across all tests? | auth | 0.878 | -0.034 | - | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question What is the recommended way to log in once and reuse the session across all tests? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: Authenticate once in a setup project, save storage state, reuse it. Actual answer The recommended approach is to authenticate once in a setup project, save the authentication state, and then reuse it to bootstrap each test already authenticated [auth#basic-shared-account-in-all-tests]. You can also reuse the signed-in state in tests by using the setup project to log in only once and skip the log-in step for all tests [best_practices#make-tests-as-isolated-as-possible~2]. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 0.500reciprocal_rank 0.333 Retrieved chunks best_practices#make-tests-as-isolated-as-possible~2rank 1 · score 0.793not in golden set await page.goto('https://github.com/login'); await page.getByLabel('Username or email address').fill('username'); await page.getByLabel('Password').fill('password'); await page.getByRole('button', { name: 'Sign in' }).click(); }); test('first', async ({ page }) => { // page is signed in. }); test('second', async ({ page }) => { // page is signed in. }); ``` You can also reuse the signed-in state in the tests with [setup project](./auth.md#basic-shared-account-in-all-tests). That way you can log in only once and then skip the log in step for all of the tests. best_practices#make-tests-as-isolated-as-possiblerank 2 · score 0.780not in golden set Each test should be completely isolated from another test and should run independently with its own local storage, session storage, data, cookies etc. [Test isolation](./browser-contexts.md) improves reproducibility, makes debugging easier and prevents cascading test failures. In order to avoid repetition for a particular part of your test you can use [before and after hooks](/api/class-test.md). Within your test file add a before hook to run a part of your test before each test such as going to a particular URL or logging in to a part of your app. This keeps your tests isolated as no test relies on another. However it is also ok to have a little duplication when tests are simple enough especially if it keeps your tests clearer and easier to read and maintain. ```js import { test } from '@playwright/test'; test.beforeEach(async ({ page }) => { // Runs before each test and signs in each page. await page.goto('https://github.com/login'); await page.getByLabel('Username or email address').fill('username'); await page.getByLabel('Password').fill('password'); await page.getByRole('button', { name: 'Sign in' }).click(); }); test('first', async ({ page }) => { // page is signed in. }); auth#basic-shared-account-in-all-testsrank 3 · score 0.778expected This is the **recommended** approach for tests **without server-side state**. Authenticate once in the **setup project**, save the authentication state, and then reuse it to bootstrap each test already authenticated. **When to use** - When you can imagine all your tests running at the same time with the same account, without affecting each other. **When not to use** - Your tests modify server-side state. For example, one test checks the rendering of the settings page, while the other test is changing the setting, and you run tests in parallel. In this case, tests must use different accounts. - Your authentication is browser-specific. **Details** Create `tests/auth.setup.ts` that will prepare authenticated browser state for all other tests. ```js import { test as setup, expect } from '@playwright/test'; import path from 'path'; const authFile = path.join(__dirname, '../playwright/.auth/user.json'); setup('authenticate', async ({ page }) => { // Perform authentication steps. Replace these actions with your own. await page.goto('https://github.com/login'); await page.getByLabel('Username or email address').fill('username'); await page.getByLabel('Password').fill('password'); await page.getByRole('button', { name: 'Sign in' }).click(); // Wait until the page receives the cookies. // // Sometimes login flow Expected but not retrieved: Score across history no pass/fail flips #1 0.83#2 0.91#3 0.88 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶fail regressed | auth-per-workerMy tests change server-side data, so a single shared login breaks them in parallel. What does the guide suggest? | auth | 0.927 | -0.073 | grounding | 1.0 | |||||||||||||||||||||||||||||||||||||||||
Question My tests change server-side data, so a single shared login breaks them in parallel. What does the guide suggest? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. required: Notes: One account per parallel worker. Actual answer The guide suggests using the "Moderate: one account per parallel worker" approach, where each parallel worker is authenticated once using a unique account [auth#moderate-one-account-per-parallel-worker]. This is recommended when tests modify shared server-side state [auth#moderate-one-account-per-parallel-worker]. Attribution generationevery expected chunk was retrieved (recall 1.0) yet the case fails on grounding. Checks
Retrieval metrics hit_rate@3 1.000precision@3 0.333recall@3 1.000reciprocal_rank 1.000 Retrieved chunks auth#moderate-one-account-per-parallel-workerrank 1 · score 0.780expected This is the **recommended** approach for tests that **modify server-side state**. In Playwright, worker processes run in parallel. In this approach, each parallel worker is authenticated once. All tests ran by worker are reusing the same authentication state. We will need multiple testing accounts, one per each parallel worker. **When to use** - Your tests modify shared server-side state. For example, one test checks the rendering of the settings page, while the other test is changing the setting. **When not to use** - Your tests do not modify any shared server-side state. In this case, all tests can use a single shared account. **Details** We will authenticate once per [worker process](./test-parallel.md#worker-processes), each with a unique account. Create `playwright/fixtures.ts` file that will [override `storageState` fixture](./test-fixtures.md#overriding-fixtures) to authenticate once per worker. Use testInfo.parallelIndex to differentiate between workers. ```js import { test as baseTest, expect } from '@playwright/test'; import fs from 'fs'; import path from 'path'; export * from '@playwright/test'; export const test = baseTest.extend<{}, { workerStorageState: string }>({ // Use the same storage state for all tests in this worker. storageState: ({ auth#authenticate-with-api-request~2rank 2 · score 0.777not in golden set workerStorageState: [async ({}, use) => { // Use parallelIndex as a unique identifier for each worker. const id = test.info().parallelIndex; const fileName = path.resolve(test.info().project.outputDir, `.auth/${id}.json`); if (fs.existsSync(fileName)) { // Reuse existing authentication state if any. await use(fileName); return; } // Important: make sure we authenticate in a clean environment by unsetting storage state. const context = await request.newContext({ storageState: undefined }); // Acquire a unique account, for example create a new one. // Alternatively, you can have a list of precreated accounts for testing. // Make sure that accounts are unique, so that multiple team members // can run tests at the same time without interference. const account = await acquireAccount(id); // Send authentication request. Replace with your own. await context.post('https://github.com/login', { form: { 'user': 'user', 'password': 'password' } }); await context.storageState({ path: fileName }); await context.dispose(); await use(fileName); }, { scope: 'worker' }], }); ``` auth#basic-shared-account-in-all-testsrank 3 · score 0.775not in golden set This is the **recommended** approach for tests **without server-side state**. Authenticate once in the **setup project**, save the authentication state, and then reuse it to bootstrap each test already authenticated. **When to use** - When you can imagine all your tests running at the same time with the same account, without affecting each other. **When not to use** - Your tests modify server-side state. For example, one test checks the rendering of the settings page, while the other test is changing the setting, and you run tests in parallel. In this case, tests must use different accounts. - Your authentication is browser-specific. **Details** Create `tests/auth.setup.ts` that will prepare authenticated browser state for all other tests. ```js import { test as setup, expect } from '@playwright/test'; import path from 'path'; const authFile = path.join(__dirname, '../playwright/.auth/user.json'); setup('authenticate', async ({ page }) => { // Perform authentication steps. Replace these actions with your own. await page.goto('https://github.com/login'); await page.getByLabel('Username or email address').fill('username'); await page.getByLabel('Password').fill('password'); await page.getByRole('button', { name: 'Sign in' }).click(); // Wait until the page receives the cookies. // // Sometimes login flow Score across history 1 flip(s), all across config changes #1 1.00#2 1.00#3 0.93 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass improved | oos-pricingHow much does the Playwright enterprise licence cost per seat? | out-of-scope | 1.000 | +0.250 | - | n/a | |||||||||||||||||||||||||||||||||||||||||
Question How much does the Playwright enterprise licence cost per seat? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. must refuse Notes: Nothing in the guides. Playwright is free and open source, but the docs here do not say so - the assistant must not invent a price. Actual answer I do not have enough information in the provided documents to answer that. Checks
Retrieval metrics hit_rate@3 n/aprecision@3 n/arecall@3 n/areciprocal_rank n/a Retrieved chunks clock#related-videosrank 1 · score 0.719not in golden set <LiteYouTube id="54_aC-rVKHg" title="Playwright 1.45" /> intro#installing-playwrightrank 2 · score 0.674not in golden set Get started by installing Playwright using one of the following methods. debug#playwright-selectorrank 3 · score 0.667not in golden set Query the Playwright selector, using the actual Playwright query engine, for example: ```bash playwright.$('.auth-form >> text=Log in'); <button>Log in</button> ``` Score across history 1 flip(s), all across config changes #1 0.75#2 0.75#3 1.00 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass improved | oos-cypress-retriesHow do I configure retries in Cypress? | out-of-scope | 1.000 | +0.250 | - | n/a | |||||||||||||||||||||||||||||||||||||||||
Question How do I configure retries in Cypress? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. must refuse Notes: hard - "retries" is a strong lexical match for the Playwright retries page. A grounded assistant must notice the question is about a different tool. Actual answer I do not have enough information in the provided documents to answer that. Checks
Retrieval metrics hit_rate@3 n/aprecision@3 n/arecall@3 n/areciprocal_rank n/a Retrieved chunks test_retries#retries~2rank 1 · score 0.714not in golden set passed (4s) ``` You can detect retries at runtime with testInfo.retry, which is accessible to any test, hook or fixture. Here is an example that clears some server-side state before a retry. ```js import { test, expect } from '@playwright/test'; test('my test', async ({ page }, testInfo) => { if (testInfo.retry) await cleanSomeCachesOnTheServer(); // ... }); ``` You can specify retries for a specific group of tests or a single file with test.describe.configure(). ```js import { test, expect } from '@playwright/test'; test.describe(() => { // All tests in this describe group will get 2 retry attempts. test.describe.configure({ retries: 2 }); test('test 1', async ({ page }) => { // ... }); test('test 2', async ({ page }) => { // ... }); }); ``` test_configuration#basic-configurationrank 2 · score 0.704not in golden set Here are some of the most common configuration options. ```js import { defineConfig, devices } from '@playwright/test'; export default defineConfig({ // Look for test files in the "tests" directory, relative to this configuration file. testDir: 'tests', // Run all tests in parallel. fullyParallel: true, // Fail the build on CI if you accidentally left test.only in the source code. forbidOnly: !!process.env.CI, // Retry on CI only. retries: process.env.CI ? 2 : 0, // Opt out of parallel tests on CI. workers: process.env.CI ? 1 : undefined, // Limit the whole test run, so that it fails with a report instead of hanging. globalTimeout: 60 * 60 * 1000, // Reporter to use reporter: 'html', use: { // Base URL to use in actions like `await page.goto('/')`. baseURL: 'http://localhost:3000', // Collect trace when retrying the failed test. trace: 'on-first-retry', }, // Configure projects for major browsers. projects: [ { name: 'chromium', use: { ...devices['Desktop Chrome'] }, }, ], // Run your local dev server before starting the tests. webServer: { command: 'npm run start', url: 'http://localhost:3000', reuseExistingServer: !process.env.CI, }, }); ``` test_global_setup_teardown#examplerank 3 · score 0.686not in golden set Here is a global setup example that authenticates once and reuses authentication state in tests. It uses the `baseURL` and `storageState` options from the configuration file. ```js import { chromium, type FullConfig } from '@playwright/test'; async function globalSetup(config: FullConfig) { const { baseURL, storageState } = config.projects[0].use; const browser = await chromium.launch(); const page = await browser.newPage(); await page.goto(baseURL!); await page.getByLabel('User Name').fill('user'); await page.getByLabel('Password').fill('password'); await page.getByText('Sign in').click(); await page.context().storageState({ path: storageState as string }); await browser.close(); } export default globalSetup; ``` Specify `globalSetup`, `baseURL` and `storageState` in the configuration file. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ globalSetup: require.resolve('./global-setup'), use: { baseURL: 'http://localhost:3000/', storageState: 'state.json', }, }); ``` Tests start already authenticated because we specify `storageState` that was populated by global setup. ```js import { test } from '@playwright/test'; test('test', async ({ page }) => { await page.goto('/'); // You are signed in! }); ``` You can make arbitrary data available in your tests from your global setup file by setting them as environment variables via `process.env`. ```js import type { FullConfig } from '@playwright/test'; async function Score across history 1 flip(s), all across config changes #1 0.74#2 0.75#3 1.00 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass improved | oos-selenium-grid-javaWrite me a Java Selenium test that logs in to my app. | out-of-scope | 1.000 | +0.250 | - | n/a | |||||||||||||||||||||||||||||||||||||||||
Question Write me a Java Selenium test that logs in to my app. Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. must refuse Notes: hard - Java samples were stripped from the corpus and Selenium is not covered by these guides. Actual answer I do not have enough information in the provided documents to answer that. Checks
Retrieval metrics hit_rate@3 n/aprecision@3 n/arecall@3 n/areciprocal_rank n/a Retrieved chunks auth#basic-shared-account-in-all-tests~3rank 1 · score 0.736not in golden set testMatch: /.*\.setup\.ts/ }, { name: 'chromium', use: { ...devices['Desktop Chrome'], // Use prepared auth state. storageState: 'playwright/.auth/user.json', }, dependencies: ['setup'], }, { name: 'firefox', use: { ...devices['Desktop Firefox'], // Use prepared auth state. storageState: 'playwright/.auth/user.json', }, dependencies: ['setup'], }, ], }); ``` Tests start already authenticated because we specified `storageState` in the config. ```js import { test } from '@playwright/test'; test('test', async ({ page }) => { // page is authenticated }); ``` Note that you need to delete the stored state when it expires. If you don't need to keep the state between test runs, write the browser state under testProject.outputDir, which is automatically cleaned up before every test run. codegen_intro#generating-locatorsrank 2 · score 0.718not in golden set You can generate [locators](/locators.md) with the test generator. * Press the `'Record'` button to stop recording and the `'Pick Locator'` button will appear * Click the `'Pick Locator'` button and hover over elements in the browser window to see the locator highlighted underneath each element * Click the element you want to locate and the code for that locator will appear in the locator playground next to the Pick Locator button * Edit the locator in the locator playground to fine-tune it and see the matching element highlighted in the browser window * Use the copy button to copy the locator and paste it into your code ######  ######  ######  ######  intro#what-s-nextrank 3 · score 0.716not in golden set - [Write tests using web-first assertions, fixtures and locators](./writing-tests.md) - [Run single or multiple tests; headed mode](./running-tests.md) - [Generate tests with Codegen](./codegen-intro.md) - [View a trace of your tests](./trace-viewer-intro.md) Score across history 1 flip(s), all across config changes #1 0.75#2 0.75#3 1.00 | |||||||||||||||||||||||||||||||||||||||||||||||
| ▶pass improved | oos-python-syntaxWhat is the Python syntax for page.get_by_role? | out-of-scope | 1.000 | +0.253 | - | n/a | |||||||||||||||||||||||||||||||||||||||||
Question What is the Python syntax for page.get_by_role? Expected No reference answer in the golden set; the case asserts on keywords, refusal and grounding. must refuse Notes: hard - the corpus is the JavaScript guide set only; the Python API is not documented here. Actual answer I do not have enough information in the provided documents to answer that. Checks
Retrieval metrics hit_rate@3 n/aprecision@3 n/arecall@3 n/areciprocal_rank n/a Retrieved chunks locators#locate-by-rolerank 1 · score 0.747not in golden set The page.getByRole() locator reflects how users and assistive technology perceive the page, for example whether some element is a button or a checkbox. When locating by role, you should usually pass the accessible name as well, so that the locator pinpoints the exact element. For example, consider the following DOM structure. ```html <h3>Sign up</h3> <label> <input type="checkbox" /> Subscribe </label> <br/> <button>Submit</button> ``` You can locate each element by its implicit role: ```js await expect(page.getByRole('heading', { name: 'Sign up' })).toBeVisible(); await page.getByRole('checkbox', { name: 'Subscribe' }).check(); await page.getByRole('button', { name: /submit/i }).click(); ``` Role locators include [buttons, checkboxes, headings, links, lists, tables, and many more](https://www.w3.org/TR/html-aria/#docconformance) and follow W3C specifications for [ARIA role](https://www.w3.org/TR/wai-aria-1.2/#roles), [ARIA attributes](https://www.w3.org/TR/wai-aria-1.2/#aria-attributes) and [accessible name](https://w3c.github.io/accname/#dfn-accessible-name). Note that many html elements like `<button>` have an [implicitly defined role](https://w3c.github.io/html-aam/#html-element-role-mappings) that is recognized by the role locator. Note that role locators **do not replace** accessibility audits and conformance tests, but rather give early feedback about the ARIA guidelines. We recommend prioritizing role locators to locate elements, as it is the closest way to how users and assistive technology perceive the page. auth#testing-multiple-roles-with-pom-fixturesrank 2 · score 0.725not in golden set **When to use** - You need to test how multiple authenticated roles interact together, in a single test. **Details** You can introduce fixtures that will provide a page authenticated as each role. Below is an example that [creates fixtures](./test-fixtures.md#creating-a-fixture) for two [Page Object Models](./pom.md) - admin POM and user POM. It assumes `adminStorageState.json` and `userStorageState.json` files were created in the global setup. ```js import { test as base, type Page, type Locator } from '@playwright/test'; // Page Object Model for the "admin" page. // Here you can add locators and helper methods specific to the admin page. class AdminPage { // Page signed in as "admin". page: Page; // Example locator pointing to "Welcome, Admin" greeting. greeting: Locator; constructor(page: Page) { this.page = page; this.greeting = page.locator('#greeting'); } } // Page Object Model for the "user" page. // Here you can add locators and helper methods specific to the user page. class UserPage { // Page signed in as "user". page: Page; // Example locator pointing to "Welcome, User" greeting. greeting: Locator; constructor(page: Page) { this.page = page; this.greeting = locators#matching-two-locators-simultaneouslyrank 3 · score 0.718not in golden set Method locator.and() narrows down an existing locator by matching an additional locator. For example, you can combine page.getByRole() and page.getByTitle() to match by both role and title. ```js const button = page.getByRole('button').and(page.getByTitle('Subscribe')); ``` Score across history 1 flip(s), all across config changes #1 0.75#2 0.75#3 1.00 | |||||||||||||||||||||||||||||||||||||||||||||||
No baseline to compare against.
Record one and re-render the dashboard; this panel then shows what regressed, degraded and improved.
ragprobe baseline ragprobe dashboardAuto-generated findings
each finding shows the rule that produced it; treat them as triage hintsbad
Weakest category: locators
3/9 passing (33.3%). Failing: loc-recommended-locator, loc-strict-mode, loc-visible-opacity, loc-stable-definition (+2 more).
Rule: categories ranked by pass rate ascending, ties broken by mean score. Only categories with at least one failing case qualify. Small categories swing hard - one case out of two is 50%.
bad
Pass rate has fallen for 2 consecutive run(s)
Now 53.2%, down from 59.7% on the previous run and 61.3% 2 runs ago.
Rule: compare pass rate on each consecutive pair of stored runs. A streak is the number of consecutive runs moving in the same direction, ending at the latest run. Runs across a config change are included, so a streak can be the sum of deliberate changes.
warn
Failures split evenly between generation and retrieval
14 a generation problem (retrieval found every expected chunk): loc-actionability-checks, loc-manual-assertion, to-global-timeout, par-retries-flag (+10 more); 14 a retrieval problem (no expected chunk was retrieved): loc-recommended-locator, loc-strict-mode, loc-visible-opacity, loc-stable-definition (+10 more); 1 mixed (retrieval was partial): cfg-default-reporter-ci.
+21 more
Rule: for each failing case, recall@k = 1.0 means every expected chunk was retrieved, so the generator had the evidence and the fault is in generation; hit rate 0 means no expected chunk was retrieved, so fix retrieval first; no expected chunks means the case tests refusal policy; anything in between is mixed; a case with no retrieval metrics is left unattributed. This is a triage hint, not a root cause.
warn
Most common failing check: keyword_presence
17 of 58 applicable cases fail keyword_presence (29.3%). Cases: loc-recommended-locator, loc-strict-mode, loc-visible-opacity, loc-stable-definition (+13 more).
+9 more
Rule: count failed, applicable checks by name over the latest run; required checks rank above advisory ones. Skipped (not applicable) checks are excluded from the denominator.
warn
The latest run followed a config change
Changed: evaluation.judge_enabled: True -> False; generation.base_url: https://api.openai.com/v1 -> https://generativelanguage.googleapis.com/v1beta/openai; generation.max_tokens: 1024 -> 2048; generation.model: claude-opus-5 -> gemini-3.5-flash-lite .... Pass rate fell from 59.7% to 53.2%.
Rule: when the latest run's config fingerprint differs from the previous run's, list the settings that differ and report the pass-rate movement across that boundary. Correlation with the change, not proof of causation.
warn
1 of 3 stored runs used a nondeterministic provider
Trend movements that include those runs are partly noise. Compare like with like: stub runs against stub runs, live runs against repeated live runs.
Rule: any stored run with deterministic = false.
info
No flaky cases; 35 case(s) are config-sensitive
No case flips under an identical config. tool-docker-image, tool-trace-on-ci, auth-per-worker, cfg-auto-fixture (+31 more) flipped only when the configuration changed, which is the harness doing its job, not noise.
+27 more
Rule: a flip is a pass/fail change between two consecutive runs that both contain the case. Flips under an identical config fingerprint are nondeterminism (flaky); flips across a config change are sensitivity to that change and are listed separately.
info
7 passing case(s) fail an advisory check
Every required check passes but at least one advisory check fails for to-default-test-timeout, cfg-auto-fixture, cfg-run-one-project, net-redirect-handler (+3 more) (most often exact_match). They clear the gate; they may still be worth a look.
Rule: passing cases with any applicable advisory check (one not in evaluation.required_checks) that failed. Advisory checks never gate, by design, so this is the only place they surface.