Playwright docs - live model vs stub baseline

Metric	Value
hit_rate@3	0.707
mrr	0.592
precision@3	0.259
recall@3	0.655

Category	Passed	Pass rate	Mean score
auth	1/2	50.0%	0.903
config	6/14	42.9%	0.683
locators	3/9	33.3%	0.627
network	5/10	50.0%	0.771
out-of-scope	4/4	100.0%	1.000
parallel	6/11	54.5%	0.678
timeouts	5/6	83.3%	0.870
tooling	3/6	50.0%	0.645

failloc-recommended-locatorWhich locator does Playwright recommend I use first when writing resilient tests?0.50

Answer

Playwright prioritizes role, text, and test id locators when figuring out the best locator [codegen#introduction, best_practices#generate-locators, debug#picking-locators].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
fail	keyword_presence	0.00	missing: getByRole
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 3 retrieved chunk(s)
fail	expected_citation_overlap	0.00	cited 0/2 expected chunk(s)
fail	grounding	0.48	1 unsupported claim(s): Playwright prioritizes role, text, and test id locators when figuring out the best locator
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	0.000
precision@3	0.000
recall@3	0.000
reciprocal_rank	0.000

Retrieved chunks

codegen#introductionrank 1 · score 0.890not in golden set

Playwright comes with the ability to generate tests for you as you perform actions in the browser and is a great way to quickly get started with testing. Playwright will look at your page and figure out the best locator, prioritizing [role, text and test id locators](./locators.md). If the generator finds multiple elements matching the locator, it will improve the locator to make it resilient that uniquely identify the target element.

best_practices#generate-locatorsrank 2 · score 0.887not in golden set

Playwright has a [test generator](./codegen.md) that can generate tests and pick locators for you. It will look at your page and figure out the best locator, prioritizing role, text and test id locators. If the generator finds multiple elements matching the locator, it will improve the locator to make it resilient and uniquely identify the target element, so you don't have to worry about failing tests due to locators.

debug#picking-locatorsrank 3 · score 0.830not in golden set

While debugging, you might need to choose a more resilient locator. You can do this by clicking on the **Pick Locator** button and hovering over any element in the browser window. While hovering over an element you will see the code needed to locate this element highlighted below. Clicking an element in the browser will add the locator into the field where you can then either tweak it or copy it into your code. <img height="1782" width="2784" alt="Picking locators" src="https://user-images.githubusercontent.com/13063165/212968640-ce82a027-9277-4bdf-b0a9-6282fb2becb7.png" /> Playwright will look at your page and figure out the best locator, prioritizing [role, text and test id locators](./locators.md). If Playwright finds multiple elements matching the locator, it will improve the locator to make it resilient and uniquely identify the target element, so you don't have to worry about failing tests due to locators.

Expected but not retrieved: locators#locate-by-role, locators#locating-elements

Faithfulness

Heuristic grounding score 0.48 over 1 claim(s).

Ungrounded claim(s):

Playwright prioritizes role, text, and test id locators when figuring out the best locator [codegen#introduction, best_practices#generate-locators, debug#picking-locators].

passloc-testid-tradeoffWhat is the downside of locating elements by test id?0.93

Answer

The downside of locating elements by test id is that test ids are not user-facing [locators#locate-by-test-id].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.64	all 1 claim(s) grounded (score 0.64)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

locators#locate-by-test-idrank 1 · score 0.821expected

Testing by test ids is the most resilient way of testing as even if your text or role of the attribute changes, the test will still pass. QA's and developers should define explicit test ids and query them with page.getByTestId(). However testing by test ids is not user facing. If the role or text value is important to you then consider using user facing locators such as [role](#locate-by-role) and [text locators](#locate-by-text). For example, consider the following DOM structure. ```html <button data-testid="directions">Itinéraire</button> ``` You can locate the element by its test id: ```js await page.getByTestId('directions').click(); ``` You can also use test ids when you choose to use the test id methodology or when you can't locate by [role](#locate-by-role) or [text](#locate-by-text).

locators#set-a-custom-test-id-attributerank 2 · score 0.792not in golden set

By default, page.getByTestId() will locate elements based on the `data-testid` attribute, but you can configure it in your test config or by calling selectors.setTestIdAttribute(). Set the test id to use a custom data attribute for your tests. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ use: { testIdAttribute: 'data-pw' } }); ``` In your html you can now use `data-pw` as your test id instead of the default `data-testid`. ```html <button data-pw="directions">Itinéraire</button> ``` And then locate the element as you would normally do: ```js await page.getByTestId('directions').click(); ```

locators#get-by-test-idrank 3 · score 0.785not in golden set

Use the page.getByTestId() method to locate an element in a list. You may need to modify the html and add a test id if you don't already have a test id. For example, consider the following DOM structure: ```html <ul> <li data-testid='apple'>apple</li> <li data-testid='banana'>banana</li> <li data-testid='orange'>orange</li> </ul> ``` Locate an item by its test id of "orange" and then click it. ```js await page.getByTestId('orange').click(); ```

Faithfulness

Heuristic grounding score 0.64 over 1 claim(s).

failloc-strict-modeMy click threw an error saying the locator resolved to more than one element. Why does Playwright do that?0.00

Answer

I do not have enough information in the provided documents to answer that.

Checks

	Check	Detail
skip	exact_match	no expected_answer in dataset
skip	fuzzy_match	no expected_answer in dataset
fail	keyword_presence	missing: strict
skip	forbidden_absent	no forbidden_keywords in dataset
fail	refusal_behaviour	refused a question it should have answered
skip	citation_present	refusals are not expected to cite sources
fail	expected_citation_overlap	cited 0/1 expected chunk(s)
skip	grounding	skipped: refusals are not grounded in retrieved text by design
skip	llm_judge_faithfulness	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	0.000
precision@3	0.000
recall@3	0.000
reciprocal_rank	0.000

Retrieved chunks

locators#throws-an-error-if-more-than-onerank 1 · score 0.869not in golden set

```js await page.getByRole('button').click(); ``` On the other hand, Playwright understands when you perform a multiple-element operation, so the following call works perfectly fine when the locator resolves to multiple elements.

debug#picking-locatorsrank 2 · score 0.836not in golden set

While debugging, you might need to choose a more resilient locator. You can do this by clicking on the **Pick Locator** button and hovering over any element in the browser window. While hovering over an element you will see the code needed to locate this element highlighted below. Clicking an element in the browser will add the locator into the field where you can then either tweak it or copy it into your code. <img height="1782" width="2784" alt="Picking locators" src="https://user-images.githubusercontent.com/13063165/212968640-ce82a027-9277-4bdf-b0a9-6282fb2becb7.png" /> Playwright will look at your page and figure out the best locator, prioritizing [role, text and test id locators](./locators.md). If Playwright finds multiple elements matching the locator, it will improve the locator to make it resilient and uniquely identify the target element, so you don't have to worry about failing tests due to locators.

locators#works-fine-with-multiple-elementsrank 3 · score 0.811not in golden set

```js await page.getByRole('button').count(); ``` You can explicitly opt-out from strictness check by telling Playwright which element to use when multiple elements match, through locator.first(), locator.last(), and locator.nth(). These methods are **not recommended** because when your page changes, Playwright may click on an element you did not intend. Instead, follow best practices above to create a locator that uniquely identifies the target element.

Expected but not retrieved: locators#strictness

Faithfulness

Heuristic grounding score 0.09 over 1 claim(s).

Ungrounded claim(s):

I do not have enough information in the provided documents to answer that.

passloc-css-vs-roleWhy shouldn't I select a button by its CSS class in my test?0.93

Answer

Your DOM can easily change, so depending on your DOM structure (such as a CSS class) can lead to failing tests if a designer changes something [best_practices#prefer-user-facing-attributes-to-xpath-or-css-selectors].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.62	all 1 claim(s) grounded (score 0.62)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

best_practices#prefer-user-facing-attributes-to-xpath-or-css-selectorsrank 1 · score 0.833expected

Your DOM can easily change so having your tests depend on your DOM structure can lead to failing tests. For example consider selecting this button by its CSS classes. Should the designer change something then the class might change, thus breaking your test. ```js // 👎 page.locator('button.buttonIcon.episode-actions-later'); ``` Use locators that are resilient to changes in the DOM. ```js // 👍 page.getByRole('button', { name: 'submit' }); ```

other_locators#css-elements-matching-one-of-the-conditionsrank 2 · score 0.790not in golden set

Comma-separated list of CSS selectors will match all elements that can be selected by one of the selectors in that list. ```js // Clicks a <button> that has either a "Log in" or "Sign in" text. await page.locator('button:has-text("Log in"), button:has-text("Sign in")').click(); ``` The `:is()` pseudo-class is a [CSS pseudo-class](https://developer.mozilla.org/en-US/docs/Web/CSS/:is) that may be useful for specifying a list of extra conditions on an element.

other_locators#css-matching-only-visible-elementsrank 3 · score 0.755not in golden set

Playwright supports the `:visible` pseudo class in CSS selectors. For example, `css=button` matches all the buttons on the page, while `css=button:visible` only matches visible buttons. This is useful to distinguish elements that are very similar but differ in visibility. Prefer the locator.visible() shortcut over the `:visible` pseudo-class. Consider a page with two buttons, first invisible and second visible. ```html <button style='display: none'>Invisible</button> <button>Visible</button> ``` * This will find both buttons and throw a [strictness](./locators.md#strictness) violation error: ```js await page.locator('button').click(); ``` ```java page.locator("button").click(); ``` ```python async await page.locator("button").click() ``` ```python sync page.locator("button").click() ``` ```csharp await page.Locator("button").ClickAsync(); ``` * This will only find a second button, because it is visible, and then click it. ```js await page.locator('button:visible').click(); ``` ```java page.locator("button:visible").click(); ``` ```python async await page.locator("button:visible").click() ``` ```python sync page.locator("button:visible").click() ``` ```csharp await page.Locator("button:visible").ClickAsync(); ```

Faithfulness

Heuristic grounding score 0.62 over 1 claim(s).

failloc-visible-opacityDoes Playwright treat an element with opacity 0 as visible?0.05

Answer

I do not have enough information in the provided documents to answer that.

Checks

	Check	Score	Detail
fail	exact_match	0.00	answer differs from expected_answer
fail	fuzzy_match	0.24	similarity 0.24 vs threshold 0.60
fail	keyword_presence	0.00	missing: opacity
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
fail	refusal_behaviour	0.00	refused a question it should have answered
skip	citation_present	0.00	refusals are not expected to cite sources
fail	expected_citation_overlap	0.00	cited 0/1 expected chunk(s)
skip	grounding	0.00	skipped: refusals are not grounded in retrieved text by design
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	0.000
precision@3	0.000
recall@3	0.000
reciprocal_rank	0.000

Retrieved chunks

other_locators#css-locatorrank 1 · score 0.815not in golden set

We recommend prioritizing [user-visible locators](./locators.md#quick-guide) like text or accessible role instead of using CSS that is tied to the implementation and could break when the page changes. Playwright can locate an element by CSS selector. ```js await page.locator('css=button').click(); ``` Playwright augments standard CSS selectors in two ways: * CSS selectors pierce open shadow DOM. * Playwright adds custom pseudo-classes like `:visible`, `:has-text()`, `:has()`, `:is()`, `:nth-match()` and more.

other_locators#css-matching-only-visible-elementsrank 2 · score 0.791not in golden set

Playwright supports the `:visible` pseudo class in CSS selectors. For example, `css=button` matches all the buttons on the page, while `css=button:visible` only matches visible buttons. This is useful to distinguish elements that are very similar but differ in visibility. Prefer the locator.visible() shortcut over the `:visible` pseudo-class. Consider a page with two buttons, first invisible and second visible. ```html <button style='display: none'>Invisible</button> <button>Visible</button> ``` * This will find both buttons and throw a [strictness](./locators.md#strictness) violation error: ```js await page.locator('button').click(); ``` ```java page.locator("button").click(); ``` ```python async await page.locator("button").click() ``` ```python sync page.locator("button").click() ``` ```csharp await page.Locator("button").ClickAsync(); ``` * This will only find a second button, because it is visible, and then click it. ```js await page.locator('button:visible').click(); ``` ```java page.locator("button:visible").click(); ``` ```python async await page.locator("button:visible").click() ``` ```python sync page.locator("button:visible").click() ``` ```csharp await page.Locator("button:visible").ClickAsync(); ```

debug#playwright-inspect-selectorrank 3 · score 0.781not in golden set

Reveal element in the Elements panel. ```bash playwright.inspect('text=Log in') ```

Expected but not retrieved: actionability#visible

Faithfulness

Heuristic grounding score 0.00 over 1 claim(s).

Ungrounded claim(s):

I do not have enough information in the provided documents to answer that.

failloc-stable-definitionWhat does Playwright mean when it waits for an element to be "stable"?0.52

Answer

Playwright considers an element to be stable when it is not animating or has completed its animation [actionability#introduction].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
fail	keyword_presence	0.00	missing: bounding box, animation frames
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
fail	expected_citation_overlap	0.00	cited 0/1 expected chunk(s)
pass	grounding	0.60	all 1 claim(s) grounded (score 0.60)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	0.000
precision@3	0.000
recall@3	0.000
reciprocal_rank	0.000

Retrieved chunks

writing_tests#assertionsrank 1 · score 0.787not in golden set

Playwright includes [test assertions](./test-assertions.md) in the form of `expect` function. To make an assertion, call `expect(value)` and choose a matcher that reflects the expectation. Playwright includes async matchers that wait until the expected condition is met. Using these matchers makes tests non-flaky and resilient. For example, this code waits until the page gets the title containing "Playwright": ```js await expect(page).toHaveTitle(/Playwright/); ``` Here are the most popular async assertions. For the complete list, see [assertions guide](./test-assertions.md): | Assertion | Description | | :- | :- | | locatorAssertions.toBeChecked() | Checkbox is checked | | locatorAssertions.toBeEnabled() | Control is enabled | | locatorAssertions.toBeVisible() | Element is visible | | locatorAssertions.toContainText() | Element contains text | | locatorAssertions.toHaveAttribute() | Element has attribute | | locatorAssertions.toHaveCount() | List of elements has given length | | locatorAssertions.toHaveText() | Element matches text | | locatorAssertions.toHaveValue() | Input element has value | | pageAssertions.toHaveTitle() | Page has title | | pageAssertions.toHaveURL() | Page has URL | Playwright also includes generic matchers like `toEqual`, `toContain`, `toBeTruthy` that can be used to assert any conditions. These assertions do not

actionability#introductionrank 2 · score 0.782not in golden set

Playwright performs a range of actionability checks on the elements before making actions to ensure these actions behave as expected. It auto-waits for all the relevant checks to pass and only then performs the requested action. If the required checks do not pass within the given `timeout`, action fails with the `TimeoutError`. For example, for locator.click(), Playwright will ensure that: - locator resolves to exactly one element - element is Visible - element is Stable, as in not animating or completed animation - element [Receives Events], as in not obscured by other elements - element is Enabled Here is the complete list of actionability checks performed for each action: | Action | Visible | Stable | [Receives Events] | Enabled | Editable | | :- | :-: | :-: | :-: | :-: | :-: | | locator.check() | Yes | Yes | Yes | Yes | - | | locator.click() | Yes | Yes | Yes | Yes | - | | locator.dblclick() | Yes | Yes | Yes | Yes | - | | locator.setChecked() | Yes |

actionability#assertionsrank 3 · score 0.779not in golden set

Expected but not retrieved: actionability#stable

Faithfulness

Heuristic grounding score 0.60 over 1 claim(s).

failloc-actionability-checksWhat checks does Playwright do automatically before clicking an element?0.92

Answer

Before clicking an element, Playwright ensures that the locator resolves to exactly one element, and that the element is visible, stable, receives events, and is enabled [actionability#introduction]. Additionally, locators perform auto-waiting to check that elements are visible and enabled before executing a click [best_practices#use-locators].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 3 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 2 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
fail	grounding	0.59	1 unsupported claim(s): Additionally, locators perform auto-waiting to check that elements are visible and enabled
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	0.333

Retrieved chunks

best_practices#use-locatorsrank 1 · score 0.842not in golden set

In order to write end to end tests we need to first find elements on the webpage. We can do this by using Playwright's built in [locators](./locators.md). Locators come with auto waiting and retry-ability. Auto waiting means that Playwright performs a range of actionability checks on the elements, such as ensuring the element is visible and enabled before it performs the click. To make tests resilient, we recommend prioritizing user-facing attributes and explicit contracts. ```js // 👍 page.getByRole('button', { name: 'submit' }); ```

debug#stepping-through-your-testsrank 2 · score 0.828not in golden set

You can play, pause or step through each action of your test using the toolbar at the top of the Inspector. You can see the current action highlighted in the test code, and matching elements highlighted in the browser window. <img height="1736" width="2680" alt="Playwright Inspector and browser" src="https://user-images.githubusercontent.com/13063165/212936618-84b87acc-bc2e-46ed-994b-32b2ef742e60.png" />

actionability#introductionrank 3 · score 0.828expected

Playwright performs a range of actionability checks on the elements before making actions to ensure these actions behave as expected. It auto-waits for all the relevant checks to pass and only then performs the requested action. If the required checks do not pass within the given `timeout`, action fails with the `TimeoutError`. For example, for locator.click(), Playwright will ensure that: - locator resolves to exactly one element - element is Visible - element is Stable, as in not animating or completed animation - element [Receives Events], as in not obscured by other elements - element is Enabled Here is the complete list of actionability checks performed for each action: | Action | Visible | Stable | [Receives Events] | Enabled | Editable | | :- | :-: | :-: | :-: | :-: | :-: | | locator.check() | Yes | Yes | Yes | Yes | - | | locator.click() | Yes | Yes | Yes | Yes | - | | locator.dblclick() | Yes | Yes | Yes | Yes | - | | locator.setChecked() | Yes |

Faithfulness

Heuristic grounding score 0.59 over 2 claim(s).

Ungrounded claim(s):

Additionally, locators perform auto-waiting to check that elements are visible and enabled before executing a click [best_practices#use-locators].

passloc-force-clickWhat does the force option on click actually skip?0.92

Answer

The `force` option disables non-essential actionability checks, such as verifying that the target element actually receives click events [actionability#forcing-actions, input#forcing-the-click].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 2 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.62	all 1 claim(s) grounded (score 0.62)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

actionability#forcing-actionsrank 1 · score 0.792expected

Some actions like locator.click() support `force` option that disables non-essential actionability checks, for example passing truthy `force` to locator.click() method will not check that the target element actually receives click events.

input#forcing-the-clickrank 2 · score 0.772not in golden set

Sometimes, apps use non-trivial logic where hovering the element overlays it with another element that intercepts the click. This behavior is indistinguishable from a bug where element gets covered and the click is dispatched elsewhere. If you know this is taking place, you can bypass the [actionability](./actionability.md) checks and force the click: ```js await page.getByRole('button').click({ force: true }); ```

input#mouse-clickrank 3 · score 0.733not in golden set

Performs a simple human click. ```js // Generic click await page.getByRole('button').click(); // Double click await page.getByText('Item').dblclick(); // Right click await page.getByText('Item').click({ button: 'right' }); // Shift + click await page.getByText('Item').click({ modifiers: ['Shift'] }); // Ctrl + click on Windows and Linux // Meta + click on macOS await page.getByText('Item').click({ modifiers: ['ControlOrMeta'] }); // Hover over element await page.getByText('Item').hover(); // Click the top left corner await page.getByText('Item').click({ position: { x: 0, y: 0 } }); ``` Under the hood, this and other pointer-related methods: - wait for element with given selector to be in DOM - wait for it to become displayed, i.e. not empty, no `display:none`, no `visibility:hidden` - wait for it to stop moving, for example, until css transition finishes - scroll the element into view - wait for it to receive pointer events at the action point, for example, waits until element becomes non-obscured by other elements - retry if the element is detached during any of the above checks

Faithfulness

Heuristic grounding score 0.62 over 1 claim(s).

failloc-manual-assertionIs expect(await locator.isVisible()).toBe(true) a good assertion?0.89

Answer

No, that is a manual assertion and is not recommended because the test will not wait for the element and will return immediately [best_practices#don-t-use-manual-assertions]. Instead, you should use web-first assertions like `await expect(locator).toBeVisible()` [best_practices#don-t-use-manual-assertions].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
fail	grounding	0.44	1 unsupported claim(s): No, that is a manual assertion and is not recommended because the test will not wait for t
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

best_practices#don-t-use-manual-assertionsrank 1 · score 0.895expected

Don't use manual assertions that are not awaiting the expect. In the code below the await is inside the expect rather than before it. When using assertions such as `isVisible()` the test won't wait a single second, it will just check the locator is there and return immediately. ```js // 👎 expect(await page.getByText('welcome').isVisible()).toBe(true); ``` Use web first assertions such as `toBeVisible()` instead. ```js // 👍 await expect(page.getByText('welcome')).toBeVisible(); ```

test_assertions#add-custom-matchers-using-expect-extend~2rank 2 · score 0.818not in golden set

${locator}\n` + `Expected: not ${this.utils.printExpected(expected)}\n` + (matcherResult ? `Received: ${this.utils.printReceived(matcherResult.actual)}` : '') : () => this.utils.matcherHint(assertionName, undefined, undefined, { isNot: this.isNot }) + '\n\n' + `Locator: ${locator}\n` + `Expected: ${this.utils.printExpected(expected)}\n` + (matcherResult ? `Received: ${this.utils.printReceived(matcherResult.actual)}` : ''); return { message, pass, name: assertionName, expected, actual: matcherResult?.actual, }; }, }); ``` Now we can use `toHaveAmount` in the test. ```js import { test, expect } from './fixtures'; test('amount', async () => { await expect(page.locator('.cart')).toHaveAmount(4); }); ```

test_assertions#custom-expect-messagerank 3 · score 0.813not in golden set

You can specify a custom expect message as a second argument to the `expect` function, for example: ```js await expect(page.getByText('Name'), 'should be logged in').toBeVisible(); ``` This message will be shown in reporters, both for passing and failing expects, providing more context about the assertion. When expect passes, you might see a successful step like this: ```txt ✅ should be logged in @example.spec.ts:18 ``` When expect fails, the error would look like this: ```bash Error: should be logged in Call log: - expect.toBeVisible with timeout 5000ms - waiting for "getByText('Name')" 2 | 3 | test('example test', async({ page }) => { > 4 | await expect(page.getByText('Name'), 'should be logged in').toBeVisible(); | ^ 5 | }); 6 | ``` Soft assertions also support custom message: ```js expect.soft(value, 'my soft assertion').toBe(56); ```

Faithfulness

Heuristic grounding score 0.44 over 2 claim(s).

Ungrounded claim(s):

No, that is a manual assertion and is not recommended because the test will not wait for the element and will return immediately [best_practices#don-t-use-manual-assertions].

passto-default-test-timeoutWhat is the default timeout for a single test?0.67

Answer

The default timeout for each test is 30 seconds [test_timeouts#test-timeout].

Checks

	Check	Score	Detail
fail	exact_match	0.00	answer differs from expected_answer
fail	fuzzy_match	0.34	similarity 0.34 vs threshold 0.60
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	0.50	cited 1/2 expected chunk(s)
pass	grounding	0.87	all 1 claim(s) grounded (score 0.87)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	0.500
reciprocal_rank	0.333

Retrieved chunks

test_timeouts#advanced-low-level-timeoutsrank 1 · score 0.810not in golden set

These are the low-level timeouts that are pre-configured by the test runner, you should not need to change these. If you happen to be in this section because your test are flaky, it is very likely that you should be looking for the solution elsewhere. |Timeout |Default |Description | |:----------|:----------------|:--------------------------------| |Action timeout| no timeout |Timeout for each action<br/><span style={{textTransform:'uppercase',fontSize:'smaller',fontWeight:'bold',opacity:'0.7'}}>Set in config</span><br/><code>{`{ use: { actionTimeout: 10_000 } }`}</code><br/><span style={{textTransform: 'uppercase',fontSize: 'smaller', fontWeight: 'bold', opacity: '0.7'}}>Override in test</span><br/>`locator.click({ timeout: 10_000 })` | |Navigation timeout| no timeout |Timeout for each navigation action<br/><span style={{textTransform:'uppercase',fontSize:'smaller',fontWeight:'bold',opacity:'0.7'}}>Set in config</span><br/><code>{`{ use: { navigationTimeout: 30_000 } }`}</code><br/><span style={{textTransform: 'uppercase',fontSize: 'smaller', fontWeight: 'bold', opacity: '0.7'}}>Override in test</span><br/>`page.goto('/', { timeout: 30_000 })` | |Global timeout|no timeout |Global timeout for the whole test run<br/><span style={{textTransform:'uppercase',fontSize:'smaller',fontWeight:'bold',opacity:'0.7'}}>Set in config</span><br/>`{ globalTimeout: 3_600_000 }`<br/> | |`beforeAll`/`afterAll` timeout|30_000 ms|Timeout for the hook<br/><span style={{textTransform:'uppercase',fontSize:'smaller',fontWeight:'bold',opacity:'0.7'}}>Set in hook</span><br/>`test.setTimeout(60_000)`<br/> | |Fixture timeout|no timeout |Timeout for an individual fixture<br/><span style={{textTransform:'uppercase',fontSize:'smaller',fontWeight:'bold',opacity:'0.7'}}>Set in fixture</span><br/>`{ scope: 'test', timeout: 30_000 }`<br/> |

test_timeouts#global-timeoutrank 2 · score 0.779not in golden set

Playwright Test supports a timeout for the whole test run. This prevents excess resource usage when everything went wrong. There is no default global timeout, but you can set a reasonable one in the config, for example one hour. Global timeout produces the following error: ```txt Running 1000 tests using 10 workers 514 skipped 486 passed Timed out waiting 3600s for the entire test run ``` You can set global timeout in the config. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ globalTimeout: 3_600_000, }); ``` API reference: testConfig.globalTimeout.

test_timeouts#test-timeoutrank 3 · score 0.763expected

Playwright Test enforces a timeout for each test, 30 seconds by default. Time spent by the test function, fixture setups, and `beforeEach` hooks is included in the test timeout. Timed out test produces the following error: ```txt example.spec.ts:3:1 › basic test =========================== Timeout of 30000ms exceeded. ``` Additional separate timeout, of the same value, is shared between fixture teardowns and `afterEach` hooks, after the test function has finished. The same timeout value also applies to `beforeAll` and `afterAll` hooks, but they do not share time with any test.

Expected but not retrieved: test_timeouts#intro

Faithfulness

Heuristic grounding score 0.87 over 1 claim(s).

passto-expect-timeoutHow long does expect(locator).toHaveText() keep retrying before it fails?0.87

Answer

`locatorAssertions.toHaveText()` has a default timeout of 5 seconds [test_timeouts#expect-timeout].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
pass	forbidden_absent	1.00	no forbidden keywords present
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	0.50	cited 1/2 expected chunk(s)
pass	grounding	0.74	all 1 claim(s) grounded (score 0.74)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	0.500
reciprocal_rank	1.000

Retrieved chunks

test_timeouts#expect-timeoutrank 1 · score 0.873expected

Auto-retrying assertions like locatorAssertions.toHaveText() have a separate timeout, 5 seconds by default. Assertion timeout is unrelated to the test timeout. It produces the following error: ```txt example.spec.ts:3:1 › basic test =========================== Error: expect(received).toHaveText(expected) Expected string: "my text" Received string: "" Call log: - expect.toHaveText with timeout 5000ms - waiting for "locator('button')" ```

test_assertions#auto-retrying-assertionsrank 2 · score 0.810not in golden set

The following assertions will retry until the assertion passes, or the assertion timeout is reached. Note that retrying assertions are async, so you must `await` them. | Assertion | Description | | :- | :- | | [await expect(locator).toBeAttached()](./api/class-locatorassertions.md#locator-assertions-to-be-attached) | Element is attached | | [await expect(locator).toBeChecked()](./api/class-locatorassertions.md#locator-assertions-to-be-checked) | Checkbox is checked | | [await expect(locator).toBeDisabled()](./api/class-locatorassertions.md#locator-assertions-to-be-disabled) | Element is disabled | | [await expect(locator).toBeEditable()](./api/class-locatorassertions.md#locator-assertions-to-be-editable) | Element is editable | | [await expect(locator).toBeEmpty()](./api/class-locatorassertions.md#locator-assertions-to-be-empty) | Container is empty | | [await expect(locator).toBeEnabled()](./api/class-locatorassertions.md#locator-assertions-to-be-enabled) | Element is enabled | | [await expect(locator).toBeFocused()](./api/class-locatorassertions.md#locator-assertions-to-be-focused) | Element is focused | | [await expect(locator).toBeHidden()](./api/class-locatorassertions.md#locator-assertions-to-be-hidden) | Element is not visible | | [await expect(locator).toBeInViewport()](./api/class-locatorassertions.md#locator-assertions-to-be-in-viewport) | Element intersects viewport | | [await expect(locator).toBeVisible()](./api/class-locatorassertions.md#locator-assertions-to-be-visible) | Element is visible | | [await expect(locator).toContainText()](./api/class-locatorassertions.md#locator-assertions-to-contain-text) | Element contains text | | [await expect(locator).toContainClass()](./api/class-locatorassertions.md#locator-assertions-to-contain-class) | Element has specified CSS classes | | [await expect(locator).toHaveAccessibleDescription()](./api/class-locatorassertions.md#locator-assertions-to-have-accessible-description) | Element has a matching [accessible description](https://w3c.github.io/accname/#dfn-accessible-description) | | [await expect(locator).toHaveAccessibleName()](./api/class-locatorassertions.md#locator-assertions-to-have-accessible-name) | Element has a matching [accessible name](https://w3c.github.io/accname/#dfn-accessible-name) | | [await expect(locator).toHaveAttribute()](./api/class-locatorassertions.md#locator-assertions-to-have-attribute) | Element has a DOM attribute | | [await expect(locator).toHaveClass()](./api/class-locatorassertions.md#locator-assertions-to-have-class) | Element has specified CSS class property | | [await

test_timeouts#specify-expect-timeout-for-a-single-assertionrank 3 · score 0.770not in golden set

```js import { test, expect } from '@playwright/test'; test('example', async ({ page }) => { await expect(locator).toHaveText('hello', { timeout: 10_000 }); }); ```

Expected but not retrieved: test_timeouts#intro

Faithfulness

Heuristic grounding score 0.74 over 1 claim(s).

failto-global-timeoutIs there a timeout for the whole test run, and what is it by default?0.86

Answer

Yes, Playwright Test supports a timeout for the whole test run, but there is no default global timeout [test_timeouts#global-timeout].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
fail	keyword_presence	0.50	missing: globalTimeout
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.81	all 1 claim(s) grounded (score 0.81)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	0.500

Retrieved chunks

test_timeouts#advanced-low-level-timeoutsrank 1 · score 0.847not in golden set

These are the low-level timeouts that are pre-configured by the test runner, you should not need to change these. If you happen to be in this section because your test are flaky, it is very likely that you should be looking for the solution elsewhere. |Timeout |Default |Description | |:----------|:----------------|:--------------------------------| |Action timeout| no timeout |Timeout for each action<br/><span style={{textTransform:'uppercase',fontSize:'smaller',fontWeight:'bold',opacity:'0.7'}}>Set in config</span><br/><code>{`{ use: { actionTimeout: 10_000 } }`}</code><br/><span style={{textTransform: 'uppercase',fontSize: 'smaller', fontWeight: 'bold', opacity: '0.7'}}>Override in test</span><br/>`locator.click({ timeout: 10_000 })` | |Navigation timeout| no timeout |Timeout for each navigation action<br/><span style={{textTransform:'uppercase',fontSize:'smaller',fontWeight:'bold',opacity:'0.7'}}>Set in config</span><br/><code>{`{ use: { navigationTimeout: 30_000 } }`}</code><br/><span style={{textTransform: 'uppercase',fontSize: 'smaller', fontWeight: 'bold', opacity: '0.7'}}>Override in test</span><br/>`page.goto('/', { timeout: 30_000 })` | |Global timeout|no timeout |Global timeout for the whole test run<br/><span style={{textTransform:'uppercase',fontSize:'smaller',fontWeight:'bold',opacity:'0.7'}}>Set in config</span><br/>`{ globalTimeout: 3_600_000 }`<br/> | |`beforeAll`/`afterAll` timeout|30_000 ms|Timeout for the hook<br/><span style={{textTransform:'uppercase',fontSize:'smaller',fontWeight:'bold',opacity:'0.7'}}>Set in hook</span><br/>`test.setTimeout(60_000)`<br/> | |Fixture timeout|no timeout |Timeout for an individual fixture<br/><span style={{textTransform:'uppercase',fontSize:'smaller',fontWeight:'bold',opacity:'0.7'}}>Set in fixture</span><br/>`{ scope: 'test', timeout: 30_000 }`<br/> |

test_timeouts#global-timeoutrank 2 · score 0.795expected

Playwright Test supports a timeout for the whole test run. This prevents excess resource usage when everything went wrong. There is no default global timeout, but you can set a reasonable one in the config, for example one hour. Global timeout produces the following error: ```txt Running 1000 tests using 10 workers 514 skipped 486 passed Timed out waiting 3600s for the entire test run ``` You can set global timeout in the config. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ globalTimeout: 3_600_000, }); ``` API reference: testConfig.globalTimeout.

ci#global-timeoutrank 3 · score 0.782not in golden set

Always set a [global timeout](./test-timeouts.md#global-timeout) in CI. By default a test run has no upper bound, so a suite that hangs, or that slowly grows past the job limit of your CI provider, is killed by the runner mid-run and does not produce the test report. With testConfig.globalTimeout set, Playwright stops the run itself. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Fail the run after an hour, so that the reporters still produce a report. globalTimeout: 60 * 60 * 1000, }); ``` The examples below therefore do not set a job-level timeout such as `timeout-minutes` in GitHub Actions. If you do add one, keep it comfortably above `globalTimeout`, so that Playwright always stops first.

Faithfulness

Heuristic grounding score 0.81 over 1 claim(s).

passto-beforeall-timeoutHow do I give a beforeAll hook more time than a normal test?0.94

Answer

You can change the timeout for a `beforeAll` hook separately by calling `testInfo.setTimeout()` inside the hook [test_timeouts#change-timeout-for-beforeall-afterall-hook].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.70	all 1 claim(s) grounded (score 0.70)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

test_timeouts#change-timeout-for-beforeall-afterall-hookrank 1 · score 0.849expected

`beforeAll` and `afterAll` hooks have a separate timeout, by default equal to test timeout. You can change it separately for each hook by calling testInfo.setTimeout() inside the hook. ```js import { test, expect } from '@playwright/test'; test.beforeAll(async () => { // Set timeout for this hook. test.setTimeout(60000); }); ``` API reference: testInfo.setTimeout().

test_parameterize#before-and-after-hooksrank 2 · score 0.783not in golden set

Most of the time you should put `beforeEach`, `beforeAll`, `afterEach` and `afterAll` hooks outside of `forEach`, so that hooks are executed just once: ```js test.beforeEach(async ({ page }) => { // ... }); test.afterEach(async ({ page }) => { // ... }); [ { name: 'Alice', expected: 'Hello, Alice!' }, { name: 'Bob', expected: 'Hello, Bob!' }, { name: 'Charlie', expected: 'Hello, Charlie!' }, ].forEach(({ name, expected }) => { test(`testing with ${name}`, async ({ page }) => { await page.goto(`https://example.com/greet?name=${name}`); await expect(page.getByRole('heading')).toHaveText(expected); }); }); ``` If you want to have hooks for each test, you can put them inside a `describe()` - so they are executed for each iteration / each individual test: ```js [ { name: 'Alice', expected: 'Hello, Alice!' }, { name: 'Bob', expected: 'Hello, Bob!' }, { name: 'Charlie', expected: 'Hello, Charlie!' }, ].forEach(({ name, expected }) => { test.describe(() => { test.beforeEach(async ({ page }) => { await page.goto(`https://example.com/greet?name=${name}`); }); test(`testing with ${expected}`, async ({ page }) => { await expect(page.getByRole('heading')).toHaveText(expected); }); }); }); ```

test_timeouts#change-timeout-from-a-beforeeach-hookrank 3 · score 0.777not in golden set

```js import { test, expect } from '@playwright/test'; test.beforeEach(async ({ page }, testInfo) => { // Extend timeout for all tests running this hook by 30 seconds. testInfo.setTimeout(testInfo.timeout + 30_000); }); ``` API reference: testInfo.setTimeout().

Faithfulness

Heuristic grounding score 0.70 over 1 claim(s).

passto-fixture-timeoutCan a slow worker fixture have its own timeout instead of eating into the test timeout?0.95

Answer

Yes, you can set a separate, larger timeout for a slow worker-scoped fixture rather than sharing the overall test timeout [test_fixtures#fixture-timeout, test_timeouts#fixture-timeout]. You can change this timeout by passing the `timeout` option to the fixture [test_fixtures#worker-scoped-fixtures].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 3 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 2/2 expected chunk(s)
pass	grounding	0.77	all 2 claim(s) grounded (score 0.77)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.667
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

test_fixtures#fixture-timeoutrank 1 · score 0.858expected

Fixture is considered to be a part of a test, and so its setup and teardown running time counts towards the test timeout. Therefore, a slow fixture may cause test timeouts. You can set a separate larger timeout for such a fixture, and keep the overall test timeout small. ```js import { test as base, expect } from '@playwright/test'; const test = base.extend<{ slowFixture: string }>({ slowFixture: [async ({}, use) => { // ... perform a slow operation ... await use('hello'); }, { timeout: 60000 }] }); test('example test', async ({ slowFixture }) => { // ... }); ``` Unlike regular test-scoped fixtures, each [worker-scoped](#worker-scoped-fixtures) fixture has its own timeout, equal to the test timeout. You can change the timeout for a worker-scoped fixture in the same way.

test_timeouts#fixture-timeoutrank 2 · score 0.854expected

By default, [fixture](./test-fixtures) shares timeout with the test. However, for slow fixtures, especially [worker-scoped](./test-fixtures#worker-scoped-fixtures) ones, it is convenient to have a separate timeout. This way you can keep the overall test timeout small, and give the slow fixture more time. ```js import { test as base, expect } from '@playwright/test'; const test = base.extend<{ slowFixture: string }>({ slowFixture: [async ({}, use) => { // ... perform a slow operation ... await use('hello'); }, { timeout: 60_000 }] }); test('example test', async ({ slowFixture }) => { // ... }); ``` API reference: test.extend().

test_fixtures#worker-scoped-fixturesrank 3 · score 0.765not in golden set

Playwright Test uses [worker processes](./test-parallel.md) to run test files. Similar to how test fixtures are set up for individual test runs, worker fixtures are set up for each worker process. That's where you can set up services, run servers, etc. Playwright Test will reuse the worker process for as many test files as it can, provided their worker fixtures match and hence environments are identical. Below we'll create an `account` fixture that will be shared by all tests in the same worker, and override the `page` fixture to log in to this account for each test. To generate unique accounts, we'll use the workerInfo.workerIndex that is available to any test or fixture. Note the tuple-like syntax for the worker fixture - we have to pass `{scope: 'worker'}` so that test runner sets this fixture up once per worker. In addition to only being run once per worker, worker-scoped fixtures also get a separate timeout equal to the default test timeout. You can change it by passing the `timeout` option. See [fixture timeout](#fixture-timeout) for more details. ```js import { test as

Faithfulness

Heuristic grounding score 0.77 over 2 claim(s).

passto-webserver-timeoutMy dev server takes ages to boot and Playwright gives up waiting for it. What do I change?0.92

Answer

You can increase the timeout by setting the `timeout` property within the `webServer` configuration in your Playwright config file [test_webserver#adding-a-server-timeout].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.57	all 1 claim(s) grounded (score 0.57)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

test_webserver#adding-a-server-timeoutrank 1 · score 0.834expected

Webservers can sometimes take longer to boot up. In this case, you can increase the timeout to wait for the server to start. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Rest of your config... // Run your local dev server before starting the tests webServer: { command: 'npm run start', url: 'http://localhost:3000', reuseExistingServer: !process.env.CI, timeout: 120 * 1000, }, }); ```

browsers#install-behind-a-firewall-or-a-proxy~2rank 2 · score 0.768not in golden set

connection timeout; it limits how long the download may go without receiving data, and defaults to 30 seconds: ```bash PLAYWRIGHT_DOWNLOAD_CONNECTION_TIMEOUT=120000 npx playwright install ``` ```bash pip install playwright PLAYWRIGHT_DOWNLOAD_CONNECTION_TIMEOUT=120000 playwright install ``` ```bash PLAYWRIGHT_DOWNLOAD_CONNECTION_TIMEOUT=120000 mvn exec:java -e -D exec.mainClass=com.microsoft.playwright.CLI -D exec.args="install" ``` ```bash PLAYWRIGHT_DOWNLOAD_CONNECTION_TIMEOUT=120000 pwsh bin/Debug/netX/playwright.ps1 install ``` If you are [installing dependencies](#install-system-dependencies) and need to use a proxy on Linux, make sure to run the command as a root user. Otherwise, Playwright will attempt to become a root and will not pass environment variables like `HTTPS_PROXY` to the linux package manager. ```bash sudo HTTPS_PROXY=https://192.0.2.1 npx playwright install-deps ``` ```bash sudo HTTPS_PROXY=https://192.0.2.1 mvn exec:java -e -D exec.mainClass=com.microsoft.playwright.CLI -D exec.args="install-deps" ``` ```bash sudo HTTPS_PROXY=https://192.0.2.1 playwright install-deps ``` ```bash sudo HTTPS_PROXY=https://192.0.2.1 pwsh bin/Debug/netX/playwright.ps1 install-deps ```

test_webserver#configuring-a-web-serverrank 3 · score 0.759not in golden set

Use the `webServer` property in your Playwright config to launch a development web server during the tests. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Run your local dev server before starting the tests webServer: { command: 'npm run start', url: 'http://localhost:3000', reuseExistingServer: !process.env.CI, stdout: 'ignore', stderr: 'pipe', }, }); ``` | Property | Description | | :- | :- | | testConfig.webServer | Launch a development web server (or multiple) during the tests. | | `command`| Shell command to start the local dev server of your app. | | `cwd` | Current working directory of the spawned process, defaults to the directory of the configuration file. | | `env` | Environment variables for the command. Defaults to inheriting `process.env` with `PLAYWRIGHT_TEST=1` added. | | `gracefulShutdown` | How to shut down the process. If unspecified, the process group is forcefully `SIGKILL`ed. If set to `{ signal: 'SIGTERM', timeout: 500 }`, the process group is sent a `SIGTERM` signal, followed by `SIGKILL` if it doesn't exit within 500ms. You can also use `SIGINT` as the signal instead. A

Faithfulness

Heuristic grounding score 0.57 over 1 claim(s).

failpar-retries-flagHow do I make Playwright retry failing tests?0.90

Answer

You can enable test retries in Playwright by passing the `--retries` command-line flag or by configuring the `retries` option in your configuration file [test_retries#retries].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
fail	grounding	0.51	grounding score 0.51 below threshold 0.55
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

test_retries#retriesrank 1 · score 0.862expected

Playwright supports **test retries**. When enabled, failing tests will be retried multiple times until they pass, or until the maximum number of retries is reached. By default failing tests are not retried. ```bash npx playwright test --retries=3 ``` You can configure retries in the configuration file: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Give failing tests 3 retry attempts retries: 3, }); ``` Playwright Test will categorize tests as follows: - "passed" - tests that passed on the first run; - "flaky" - tests that failed on the first run, but passed when retried; - "failed" - tests that failed on the first run and failed all retries. ```bash Running 3 tests using 1 worker ✓ example.spec.ts:4:2 › first passes (438ms) x example.spec.ts:5:2 › second flaky (691ms) ✓ example.spec.ts:5:2 › second flaky (522ms) ✓ example.spec.ts:6:2 › third passes (932ms) 1 flaky example.spec.ts:5:2 › second flaky 2 passed (4s) ``` You can detect retries at runtime with testInfo.retry, which is accessible to any test, hook or fixture. Here is an example that clears some server-side state before

test_parallel#limit-failures-and-fail-fastrank 2 · score 0.841not in golden set

You can limit the number of failed tests in the whole test suite by setting `maxFailures` config option or passing `--max-failures` command line flag. When running with "max failures" set, Playwright Test will stop after reaching this number of failed tests and skip any tests that were not executed yet. This is useful to avoid wasting resources on broken test suites. Passing command line option: ```bash npx playwright test --max-failures=10 ``` Setting in the configuration file: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Limit the number of failures on CI to save resources maxFailures: process.env.CI ? 10 : undefined, }); ```

ci_intro#viewing-the-tracerank 3 · score 0.822not in golden set

Once you have served the report using `npx playwright show-report`, click on the trace icon next to the test's file name as seen in the image above. You can then view the trace of your tests and inspect each action to try to find out why the tests are failing. <img src="https://github.com/microsoft/playwright/assets/13063165/10fe3585-8401-4051-b1c2-b2e92ac4c274" alt="playwright trace viewer" width="3598" height="2218" />

Faithfulness

Heuristic grounding score 0.51 over 1 claim(s).

passpar-flaky-definitionWhat does Playwright count as a "flaky" test in its results?0.93

Answer

Playwright categorizes a test as "flaky" if it failed on the first run, but passed when retried [test_retries#retries].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 2 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.65	all 1 claim(s) grounded (score 0.65)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

test_retries#retriesrank 1 · score 0.778expected

Playwright supports **test retries**. When enabled, failing tests will be retried multiple times until they pass, or until the maximum number of retries is reached. By default failing tests are not retried. ```bash npx playwright test --retries=3 ``` You can configure retries in the configuration file: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Give failing tests 3 retry attempts retries: 3, }); ``` Playwright Test will categorize tests as follows: - "passed" - tests that passed on the first run; - "flaky" - tests that failed on the first run, but passed when retried; - "failed" - tests that failed on the first run and failed all retries. ```bash Running 3 tests using 1 worker ✓ example.spec.ts:4:2 › first passes (438ms) x example.spec.ts:5:2 › second flaky (691ms) ✓ example.spec.ts:5:2 › second flaky (522ms) ✓ example.spec.ts:6:2 › third passes (932ms) 1 flaky example.spec.ts:5:2 › second flaky 2 passed (4s) ``` You can detect retries at runtime with testInfo.retry, which is accessible to any test, hook or fixture. Here is an example that clears some server-side state before

writing_tests#assertionsrank 2 · score 0.770not in golden set

Playwright includes [test assertions](./test-assertions.md) in the form of `expect` function. To make an assertion, call `expect(value)` and choose a matcher that reflects the expectation. Playwright includes async matchers that wait until the expected condition is met. Using these matchers makes tests non-flaky and resilient. For example, this code waits until the page gets the title containing "Playwright": ```js await expect(page).toHaveTitle(/Playwright/); ``` Here are the most popular async assertions. For the complete list, see [assertions guide](./test-assertions.md): | Assertion | Description | | :- | :- | | locatorAssertions.toBeChecked() | Checkbox is checked | | locatorAssertions.toBeEnabled() | Control is enabled | | locatorAssertions.toBeVisible() | Element is visible | | locatorAssertions.toContainText() | Element contains text | | locatorAssertions.toHaveAttribute() | Element has attribute | | locatorAssertions.toHaveCount() | List of elements has given length | | locatorAssertions.toHaveText() | Element matches text | | locatorAssertions.toHaveValue() | Input element has value | | pageAssertions.toHaveTitle() | Page has title | | pageAssertions.toHaveURL() | Page has URL | Playwright also includes generic matchers like `toEqual`, `toContain`, `toBeTruthy` that can be used to assert any conditions. These assertions do not

test_parallel#avoiding-shared-state-in-parallel-testsrank 3 · score 0.756not in golden set

Playwright runs tests in separate worker processes, each with its own isolated BrowserContext, so cookies, storage and in-memory globals are already isolated. Flakiness comes from state that lives *outside* a single test. Here are recipes for the common cases.

Faithfulness

Heuristic grounding score 0.65 over 1 claim(s).

failpar-retries-defaultAre failing tests retried by default?0.04

Answer

I do not have enough information in the provided documents to answer that.

Checks

	Check	Score	Detail
fail	exact_match	0.00	answer differs from expected_answer
fail	fuzzy_match	0.19	similarity 0.19 vs threshold 0.60
fail	keyword_presence	0.00	missing: not retried
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
fail	refusal_behaviour	0.00	refused a question it should have answered
skip	citation_present	0.00	refusals are not expected to cite sources
fail	expected_citation_overlap	0.00	cited 0/1 expected chunk(s)
skip	grounding	0.00	skipped: refusals are not grounded in retrieved text by design
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	0.000
precision@3	0.000
recall@3	0.000
reciprocal_rank	0.000

Retrieved chunks

test_retries#introductionrank 1 · score 0.759not in golden set

Test retries are a way to automatically re-run a test when it fails. This is useful when a test is flaky and fails intermittently. Test retries are configured in the [configuration file](./test-configuration.md).

browser_contexts#why-is-test-isolation-importantrank 2 · score 0.737not in golden set

- No failure carry-over. If one test fails it doesn't affect the other test. - Easy to debug errors or flakiness, because you can run just a single test as many times as you'd like. - Don't have to think about the order when running in parallel, sharding, etc.

test_retries#failures~2rank 3 · score 0.737not in golden set

with the browser and will start a new one. Testing will continue in the new worker process starting with the next test. * Worker process #1 starts * `beforeAll` hook runs * `first good` passes * `second flaky` fails * `afterAll` hook runs * Worker process #2 starts * `beforeAll` hook runs again * `third good` passes * `afterAll` hook runs If you **enable [retries](#retries)**, second worker process will start by retrying the failed test and continue from there. * Worker process #1 starts * `beforeAll` hook runs * `first good` passes * `second flaky` fails * `afterAll` hook runs * Worker process #2 starts * `beforeAll` hook runs again * `second flaky` is retried and passes * `third good` passes * `afterAll` hook runs This scheme works perfectly for independent tests and guarantees that failing tests can't affect healthy ones.

Expected but not retrieved: test_retries#retries

Faithfulness

Heuristic grounding score 0.00 over 1 claim(s).

Ungrounded claim(s):

I do not have enough information in the provided documents to answer that.

passpar-workers-cliHow do I limit the number of parallel workers on CI but keep the default locally?0.95

Answer

You can limit the number of workers on CI while using the default locally by setting `workers: process.env.CI ? 2 : undefined` (or `1` for CI) in your configuration file [test_parallel#limit-workers, ci#workers].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 2 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 2 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.74	all 2 claim(s) grounded (score 0.74)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

test_parallel#limit-workersrank 1 · score 0.867expected

You can control the maximum number of parallel worker processes via [command line](./test-cli.md) or in the [configuration file](./test-configuration.md). From the command line: ```bash npx playwright test --workers 4 ``` In the configuration file: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Limit the number of workers on CI, use default locally workers: process.env.CI ? 2 : undefined, }); ```

ci#workersrank 2 · score 0.796not in golden set

We recommend setting [workers](./api/class-testconfig.md#test-config-workers) to "1" in CI environments to prioritize stability and reproducibility. Running tests sequentially ensures each test gets the full system resources, avoiding potential conflicts. However, if you have a powerful self-hosted CI system, you may enable [parallel](./test-parallel.md) tests. For wider parallelization, consider [sharding](./test-parallel.md#shard-tests-between-multiple-machines) - distributing tests across multiple CI jobs. ```js import { defineConfig, devices } from '@playwright/test'; export default defineConfig({ // Opt out of parallel tests on CI. workers: process.env.CI ? 1 : undefined, }); ```

test_parallel#disable-parallelismrank 3 · score 0.791not in golden set

You can disable any parallelism by allowing just a single worker at any time. Either set `workers: 1` option in the configuration file or pass `--workers=1` to the command line. ```bash npx playwright test --workers=1 ```

Faithfulness

Heuristic grounding score 0.74 over 2 claim(s).

passpar-disable-parallelHow do I turn parallelism off completely?0.93

Answer

You can disable parallelism completely by setting the `workers: 1` option in the configuration file or by passing `--workers=1` to the command line [test_parallel#disable-parallelism].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.67	all 1 claim(s) grounded (score 0.67)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

test_parallel#disable-parallelismrank 1 · score 0.801expected

You can disable any parallelism by allowing just a single worker at any time. Either set `workers: 1` option in the configuration file or pass `--workers=1` to the command line. ```bash npx playwright test --workers=1 ```

test_parallel#limit-workersrank 2 · score 0.699not in golden set

You can control the maximum number of parallel worker processes via [command line](./test-cli.md) or in the [configuration file](./test-configuration.md). From the command line: ```bash npx playwright test --workers 4 ``` In the configuration file: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Limit the number of workers on CI, use default locally workers: process.env.CI ? 2 : undefined, }); ```

test_parallel#sort-test-files-alphabeticallyrank 3 · score 0.693not in golden set

When you **disable parallel test execution**, Playwright Test runs test files in alphabetical order. You can use some naming convention to control the test order, for example `001-user-signin-flow.spec.ts`, `002-create-new-document.spec.ts` and so on.

Faithfulness

Heuristic grounding score 0.67 over 1 claim(s).

failpar-serial-modeMy tests depend on each other. How do I make them run in order and stop after the first failure?0.53

Answer

To run test files in order, you must disable parallel test execution so that Playwright Test runs them alphabetically [test_parallel#sort-test-files-alphabetically]. To stop the test suite after failures, you can set the `maxFailures` config option or pass the `--max-failures` command-line flag [test_parallel#limit-failures-and-fail-fast].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
fail	keyword_presence	0.00	missing: serial
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 2 retrieved chunk(s)
fail	expected_citation_overlap	0.00	cited 0/2 expected chunk(s)
pass	grounding	0.63	all 2 claim(s) grounded (score 0.63)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	0.000
precision@3	0.000
recall@3	0.000
reciprocal_rank	0.000

Retrieved chunks

test_parallel#sort-test-files-alphabeticallyrank 1 · score 0.784not in golden set

When you **disable parallel test execution**, Playwright Test runs test files in alphabetical order. You can use some naming convention to control the test order, for example `001-user-signin-flow.spec.ts`, `002-create-new-document.spec.ts` and so on.

test_parallel#parallelize-tests-in-a-single-filerank 2 · score 0.775not in golden set

By default, tests in a single file are run in order. If you have many independent tests in a single file, you might want to run them in parallel with test.describe.configure(). Note that parallel tests are executed in separate worker processes and cannot share any state or global variables. Each test executes all relevant hooks just for itself, including `beforeAll` and `afterAll`. ```js import { test } from '@playwright/test'; test.describe.configure({ mode: 'parallel' }); test('runs in parallel 1', async ({ page }) => { /* ... */ }); test('runs in parallel 2', async ({ page }) => { /* ... */ }); ``` Alternatively, you can opt-in all tests into this fully-parallel mode in the configuration file: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ fullyParallel: true, }); ``` You can also opt in for fully-parallel mode for just a few projects: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // runs all tests in all files of a specific project in parallel projects: [ { name: 'chromium', use: { ...devices['Desktop Chrome'] }, fullyParallel: true, }, ]

test_parallel#limit-failures-and-fail-fastrank 3 · score 0.766not in golden set

You can limit the number of failed tests in the whole test suite by setting `maxFailures` config option or passing `--max-failures` command line flag. When running with "max failures" set, Playwright Test will stop after reaching this number of failed tests and skip any tests that were not executed yet. This is useful to avoid wasting resources on broken test suites. Passing command line option: ```bash npx playwright test --max-failures=10 ``` Setting in the configuration file: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Limit the number of failures on CI to save resources maxFailures: process.env.CI ? 10 : undefined, }); ```

Expected but not retrieved: test_parallel#serial-mode, test_retries#serial-mode

Faithfulness

Heuristic grounding score 0.63 over 2 claim(s).

passpar-max-failuresCan I stop the whole run after ten failures so I don't waste CI minutes?0.95

Answer

Yes, you can limit the number of failed tests by setting the `maxFailures` config option or passing the `--max-failures` command line flag [test_parallel#limit-failures-and-fail-fast]. For example, you can set it to 10 to stop the test suite after reaching that number of failures [test_parallel#limit-failures-and-fail-fast].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.76	all 2 claim(s) grounded (score 0.76)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	0.333

Retrieved chunks

test_use_options#trace-modes~2rank 1 · score 0.766not in golden set

test_use_options#video-modes~2rank 2 · score 0.748not in golden set

:- | :- | :- | | `'off'` | — | — | — | | `'on'` | first run | first run + retry | all three runs | | `'retain-on-failure'` | — | first run | all three runs | | `'retain-on-first-failure'` | — | first run | first run | | `'retain-on-failure-and-retries'` | — | first run + retry | all three runs | | `'on-first-retry'` | — | first retry | first retry | | `'on-all-retries'` | — | first retry | both retries |

test_parallel#limit-failures-and-fail-fastrank 3 · score 0.726expected

You can limit the number of failed tests in the whole test suite by setting `maxFailures` config option or passing `--max-failures` command line flag. When running with "max failures" set, Playwright Test will stop after reaching this number of failed tests and skip any tests that were not executed yet. This is useful to avoid wasting resources on broken test suites. Passing command line option: ```bash npx playwright test --max-failures=10 ``` Setting in the configuration file: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Limit the number of failures on CI to save resources maxFailures: process.env.CI ? 10 : undefined, }); ```

Faithfulness

Heuristic grounding score 0.76 over 2 claim(s).

failpar-shard-flagHow do I split my suite across four CI machines?0.00

Answer

I do not have enough information in the provided documents to answer that.

Checks

	Check	Detail
skip	exact_match	no expected_answer in dataset
skip	fuzzy_match	no expected_answer in dataset
fail	keyword_presence	missing: --shard
skip	forbidden_absent	no forbidden_keywords in dataset
fail	refusal_behaviour	refused a question it should have answered
skip	citation_present	refusals are not expected to cite sources
fail	expected_citation_overlap	cited 0/1 expected chunk(s)
skip	grounding	skipped: refusals are not grounded in retrieved text by design
skip	llm_judge_faithfulness	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	0.000
precision@3	0.000
recall@3	0.000
reciprocal_rank	0.000

Retrieved chunks

ci#ci-configurationsrank 1 · score 0.690not in golden set

The [Command line tools](./browsers#install-system-dependencies) can be used to install all operating system dependencies in CI.

test_parallel#limit-workersrank 2 · score 0.675not in golden set

You can control the maximum number of parallel worker processes via [command line](./test-cli.md) or in the [configuration file](./test-configuration.md). From the command line: ```bash npx playwright test --workers 4 ``` In the configuration file: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Limit the number of workers on CI, use default locally workers: process.env.CI ? 2 : undefined, }); ```

ci#workersrank 3 · score 0.663not in golden set

We recommend setting [workers](./api/class-testconfig.md#test-config-workers) to "1" in CI environments to prioritize stability and reproducibility. Running tests sequentially ensures each test gets the full system resources, avoiding potential conflicts. However, if you have a powerful self-hosted CI system, you may enable [parallel](./test-parallel.md) tests. For wider parallelization, consider [sharding](./test-parallel.md#shard-tests-between-multiple-machines) - distributing tests across multiple CI jobs. ```js import { defineConfig, devices } from '@playwright/test'; export default defineConfig({ // Opt out of parallel tests on CI. workers: process.env.CI ? 1 : undefined, }); ```

Expected but not retrieved: test_sharding#sharding-tests-between-multiple-machines

Faithfulness

Heuristic grounding score 0.00 over 1 claim(s).

Ungrounded claim(s):

I do not have enough information in the provided documents to answer that.

passpar-merge-reportsAfter sharding, how do I get one HTML report instead of four?0.86

Answer

To get one HTML report, put the blob report files from all shards into a single directory and run `npx playwright merge-reports --reporter html ./all-blob-reports` [test_sharding#merging-reports-from-multiple-shards].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	0.50	cited 1/2 expected chunk(s)
pass	grounding	0.78	all 1 claim(s) grounded (score 0.78)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	0.500
reciprocal_rank	1.000

Retrieved chunks

test_sharding#merging-reports-from-multiple-shardsrank 1 · score 0.762expected

In the previous example, each test shard has its own test report. If you want to have a combined report showing all the test results from all the shards, you can merge them. Start with adding `blob` reporter to the config when running on CI: ```js export default defineConfig({ testDir: './tests', reporter: process.env.CI ? 'blob' : 'html', }); ``` Blob report contains information about all the tests that were run and their results as well as all test attachments such as traces and screenshot diffs. Blob reports can be merged and converted to any other Playwright report. By default, blob report will be generated into `blob-report` directory. You can learn about [blob report options here](./test-reporters.md#blob-reporter). To merge reports from multiple shards, put the blob report files into a single directory, for example `all-blob-reports`. Blob report names contain shard number, so they will not clash. Afterwards, run `npx playwright merge-reports` command: ```bash npx playwright merge-reports --reporter html ./all-blob-reports ``` This will produce a standard HTML report into `playwright-report` directory.

test_cli#show-reportrank 2 · score 0.751not in golden set

Display HTML report from previous test run. [Read more about the HTML reporter](./test-reporters#html-reporter).

test_reporters#blob-reporter~2rank 3 · score 0.745not in golden set

blob report to merge multiple shards, you don't have to pass any options. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ reporter: 'blob', }); ``` </TabItem> <TabItem value="environments"> When running tests in different environments, you might want to use testConfig.tag to add a global tag corresponding to the environment. This tag will bring clarity to the merged report, and it will be used to produce a unique blob report name. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ reporter: 'blob', tag: process.env.CI_ENVIRONMENT_NAME, // for example "@APIv2" or "@linux" }); ``` </TabItem> </Tabs> Blob report supports following configuration options and environment variables: | Environment Variable Name | Reporter Config Option| Description | Default |---|---|---|---| | `PLAYWRIGHT_BLOB_OUTPUT_DIR` | `outputDir` | Directory to save the output. Existing content is deleted before writing the new report. | `blob-report` | `PLAYWRIGHT_BLOB_OUTPUT_NAME` | `fileName` | Report file name. | `report-<hash>-<shard_number>.zip` | `PLAYWRIGHT_BLOB_OUTPUT_FILE` | `outputFile` | Full path to the output file. If defined, `outputDir` and `fileName` will be ignored. | `undefined`

Expected but not retrieved: test_sharding#merge-reports-cli

Faithfulness

Heuristic grounding score 0.78 over 1 claim(s).

passpar-worker-indexHow can a test find out which worker it is running in, so each worker uses its own test account?0.81

Answer

To differentiate between workers and generate unique accounts, tests and fixtures can use `testInfo.workerIndex` or `workerInfo.workerIndex` [test_parallel#isolate-test-data-between-parallel-workers, test_fixtures#worker-scoped-fixtures]. Additionally, `test.info().parallelIndex` can be used as a unique identifier for a worker [auth#authenticate-with-api-request~2].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 3 retrieved chunk(s)
pass	expected_citation_overlap	0.50	cited 1/2 expected chunk(s)
pass	grounding	0.57	all 2 claim(s) grounded (score 0.57)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	0.500
reciprocal_rank	1.000

Retrieved chunks

test_parallel#isolate-test-data-between-parallel-workersrank 1 · score 0.787expected

You can leverage `process.env.TEST_WORKER_INDEX` or testInfo.workerIndex mentioned above to isolate user data in the database between tests running on different workers. All tests run by the worker reuse the same user. Create `playwright/fixtures.ts` file that will [create `dbUserName` fixture](./test-fixtures#creating-a-fixture) and initialize a new user in the test database. Use testInfo.workerIndex to differentiate between workers. ```js import { test as baseTest, expect } from '@playwright/test'; // Import project utils for managing users in the test database. import { createUserInTestDatabase, deleteUserFromTestDatabase } from './my-db-utils'; export * from '@playwright/test'; export const test = baseTest.extend<{}, { dbUserName: string }>({ // Returns db user name unique for the worker. dbUserName: [async ({ }, use) => { // Use workerIndex as a unique identifier for each worker. const userName = `user-${test.info().workerIndex}`; // Initialize user in the database. await createUserInTestDatabase(userName); await use(userName); // Clean up after the tests are done. await deleteUserFromTestDatabase(userName); }, { scope: 'worker' }], }); ``` Now, each test file should import `test` from our fixtures file instead of `@playwright/test`. ```js // Important: import our fixtures. import { test, expect } from '../playwright/fixtures'; test('test',

auth#authenticate-with-api-request~2rank 2 · score 0.763not in golden set

workerStorageState: [async ({}, use) => { // Use parallelIndex as a unique identifier for each worker. const id = test.info().parallelIndex; const fileName = path.resolve(test.info().project.outputDir, `.auth/${id}.json`); if (fs.existsSync(fileName)) { // Reuse existing authentication state if any. await use(fileName); return; } // Important: make sure we authenticate in a clean environment by unsetting storage state. const context = await request.newContext({ storageState: undefined }); // Acquire a unique account, for example create a new one. // Alternatively, you can have a list of precreated accounts for testing. // Make sure that accounts are unique, so that multiple team members // can run tests at the same time without interference. const account = await acquireAccount(id); // Send authentication request. Replace with your own. await context.post('https://github.com/login', { form: { 'user': 'user', 'password': 'password' } }); await context.storageState({ path: fileName }); await context.dispose(); await use(fileName); }, { scope: 'worker' }], }); ```

test_fixtures#worker-scoped-fixturesrank 3 · score 0.757not in golden set

Playwright Test uses [worker processes](./test-parallel.md) to run test files. Similar to how test fixtures are set up for individual test runs, worker fixtures are set up for each worker process. That's where you can set up services, run servers, etc. Playwright Test will reuse the worker process for as many test files as it can, provided their worker fixtures match and hence environments are identical. Below we'll create an `account` fixture that will be shared by all tests in the same worker, and override the `page` fixture to log in to this account for each test. To generate unique accounts, we'll use the workerInfo.workerIndex that is available to any test or fixture. Note the tuple-like syntax for the worker fixture - we have to pass `{scope: 'worker'}` so that test runner sets this fixture up once per worker. In addition to only being run once per worker, worker-scoped fixtures also get a separate timeout equal to the default test timeout. You can change it by passing the `timeout` option. See [fixture timeout](#fixture-timeout) for more details. ```js import { test as

Expected but not retrieved: test_parallel#worker-index-and-parallel-index

Faithfulness

Heuristic grounding score 0.57 over 2 claim(s).

failpar-isolation-contextHow does Playwright keep one test's cookies and local storage from leaking into the next test?0.56

Answer

Playwright runs tests in separate worker processes, each equipped with its own isolated BrowserContext to isolate cookies, storage, and in-memory globals [test_parallel#avoiding-shared-state-in-parallel-tests].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
fail	keyword_presence	0.00	missing: browser context
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
fail	expected_citation_overlap	0.00	cited 0/2 expected chunk(s)
pass	grounding	0.78	all 1 claim(s) grounded (score 0.78)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	0.000
precision@3	0.000
recall@3	0.000
reciprocal_rank	0.000

Retrieved chunks

auth#avoid-authentication-in-some-testsrank 1 · score 0.855not in golden set

You can reset storage state in a test file to avoid authentication that was set up for the whole project. ```js import { test } from '@playwright/test'; // Reset storage state for this file to avoid being authenticated test.use({ storageState: { cookies: [], origins: [] } }); test('not signed in test', async ({ page }) => { // ... }); ```

test_parallel#avoiding-shared-state-in-parallel-testsrank 2 · score 0.803not in golden set

Playwright runs tests in separate worker processes, each with its own isolated BrowserContext, so cookies, storage and in-memory globals are already isolated. Flakiness comes from state that lives *outside* a single test. Here are recipes for the common cases.

test_global_setup_teardown#examplerank 3 · score 0.797not in golden set

Here is a global setup example that authenticates once and reuses authentication state in tests. It uses the `baseURL` and `storageState` options from the configuration file. ```js import { chromium, type FullConfig } from '@playwright/test'; async function globalSetup(config: FullConfig) { const { baseURL, storageState } = config.projects[0].use; const browser = await chromium.launch(); const page = await browser.newPage(); await page.goto(baseURL!); await page.getByLabel('User Name').fill('user'); await page.getByLabel('Password').fill('password'); await page.getByText('Sign in').click(); await page.context().storageState({ path: storageState as string }); await browser.close(); } export default globalSetup; ``` Specify `globalSetup`, `baseURL` and `storageState` in the configuration file. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ globalSetup: require.resolve('./global-setup'), use: { baseURL: 'http://localhost:3000/', storageState: 'state.json', }, }); ``` Tests start already authenticated because we specify `storageState` that was populated by global setup. ```js import { test } from '@playwright/test'; test('test', async ({ page }) => { await page.goto('/'); // You are signed in! }); ``` You can make arbitrary data available in your tests from your global setup file by setting them as environment variables via `process.env`. ```js import type { FullConfig } from '@playwright/test'; async function

Expected but not retrieved: browser_contexts#how-playwright-achieves-test-isolation, browser_contexts#introduction

Faithfulness

Heuristic grounding score 0.78 over 1 claim(s).

failcfg-worker-fixtureHow do I set something up once per worker instead of once per test?0.00

Answer

I do not have enough information in the provided documents to answer that.

Checks

	Check	Detail
skip	exact_match	no expected_answer in dataset
skip	fuzzy_match	no expected_answer in dataset
fail	keyword_presence	missing: scope: 'worker'
skip	forbidden_absent	no forbidden_keywords in dataset
fail	refusal_behaviour	refused a question it should have answered
skip	citation_present	refusals are not expected to cite sources
fail	expected_citation_overlap	cited 0/1 expected chunk(s)
skip	grounding	skipped: refusals are not grounded in retrieved text by design
skip	llm_judge_faithfulness	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	0.000
precision@3	0.000
recall@3	0.000
reciprocal_rank	0.000

Retrieved chunks

ci#workersrank 1 · score 0.826not in golden set

We recommend setting [workers](./api/class-testconfig.md#test-config-workers) to "1" in CI environments to prioritize stability and reproducibility. Running tests sequentially ensures each test gets the full system resources, avoiding potential conflicts. However, if you have a powerful self-hosted CI system, you may enable [parallel](./test-parallel.md) tests. For wider parallelization, consider [sharding](./test-parallel.md#shard-tests-between-multiple-machines) - distributing tests across multiple CI jobs. ```js import { defineConfig, devices } from '@playwright/test'; export default defineConfig({ // Opt out of parallel tests on CI. workers: process.env.CI ? 1 : undefined, }); ```

test_parallel#limit-workersrank 2 · score 0.797not in golden set

You can control the maximum number of parallel worker processes via [command line](./test-cli.md) or in the [configuration file](./test-configuration.md). From the command line: ```bash npx playwright test --workers 4 ``` In the configuration file: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Limit the number of workers on CI, use default locally workers: process.env.CI ? 2 : undefined, }); ```

getting_started_vscode#global-setuprank 3 · score 0.782not in golden set

For tasks that need to run only once before all tests (like seeding a database), use **Global Setup**. You can trigger the global setup and teardown manually from the Playwright sidebar. ![running global setup](./images/getting-started/global-setup.png)

Expected but not retrieved: test_fixtures#worker-scoped-fixtures

Faithfulness

Heuristic grounding score 0.00 over 1 claim(s).

Ungrounded claim(s):

I do not have enough information in the provided documents to answer that.

passcfg-auto-fixtureCan a fixture run for every test without each test having to list it as a parameter?0.72

Answer

Yes, you can declare them as auto fixtures with `scope: 'worker'` and `auto: true` so they automatically run for every worker without being redeclared in every file [test_fixtures#adding-global-beforeall-afterall-hooks].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
fail	expected_citation_overlap	0.00	cited 0/1 expected chunk(s)
pass	grounding	0.62	all 1 claim(s) grounded (score 0.62)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	0.000
precision@3	0.000
recall@3	0.000
reciprocal_rank	0.000

Retrieved chunks

test_fixtures#combine-custom-fixtures-from-multiple-modulesrank 1 · score 0.784not in golden set

You can merge test fixtures from multiple files or modules: ```js import { mergeTests } from '@playwright/test'; import { test as dbTest } from 'database-test-utils'; import { test as a11yTest } from 'a11y-test-utils'; export const test = mergeTests(dbTest, a11yTest); ``` ```js import { test } from './fixtures'; test('passes', async ({ database, page, a11y }) => { // use database and a11y fixtures. }); ```

test_fixtures#using-a-fixturerank 2 · score 0.783not in golden set

Just mention a fixture in your test function argument, and the test runner will take care of it. Fixtures are also available in hooks and other fixtures. If you use TypeScript, fixtures will be type safe. Below we use the `todoPage` and `settingsPage` fixtures that we defined above. ```js import { test, expect } from './my-test'; test.beforeEach(async ({ settingsPage }) => { await settingsPage.switchToDarkMode(); }); test('basic test', async ({ todoPage, page }) => { await todoPage.addToDo('something nice'); await expect(page.getByTestId('todo-title')).toContainText(['something nice']); }); ```

test_fixtures#adding-global-beforeall-afterall-hooksrank 3 · score 0.777not in golden set

test.beforeAll() and test.afterAll() hooks run before/after all tests declared in the same file and same test.describe() block (if any), once per worker process. If you want to declare hooks that run before/after all tests in every file, you can declare them as auto fixtures with `scope: 'worker'` as follows: ```js import { test as base } from '@playwright/test'; export const test = base.extend<{}, { forEachWorker: void }>({ forEachWorker: [async ({}, use) => { // This code runs before all the tests in the worker process. console.log(`Starting test worker ${test.info().workerIndex}`); await use(); // This code runs after all the tests in the worker process. console.log(`Stopping test worker ${test.info().workerIndex}`); }, { scope: 'worker', auto: true }], // automatically starts for every worker. }); ``` And then import the fixtures in all your tests: ```js import { test } from './fixtures'; import { expect } from '@playwright/test'; test('basic', async ({ }) => { // ... }); ``` Note that the fixtures will still run once per [worker process](./test-parallel.md#worker-processes), but you don't need to redeclare them in every file.

Expected but not retrieved: test_fixtures#automatic-fixtures

Faithfulness

Heuristic grounding score 0.62 over 1 claim(s).

passcfg-fixture-orderIf fixture A depends on fixture B, which one is torn down first?0.93

Answer

Fixture A is torn down first, while fixture B is torn down after A [test_fixtures#execution-order].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.64	all 1 claim(s) grounded (score 0.64)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

test_fixtures#execution-orderrank 1 · score 0.760expected

Each fixture has a setup and teardown phase before and after the `await use()` call in the fixture. Setup is executed before the test/hook requiring it is run, and teardown is executed when the fixture is no longer being used by the test/hook. Fixtures follow these rules to determine the execution order: * When fixture A depends on fixture B: B is always set up before A and torn down after A. * Non-automatic fixtures are executed lazily, only when the test/hook needs them. * Test-scoped fixtures are torn down after each test, while worker-scoped fixtures are only torn down when the worker process executing tests is torn down. Consider the following example: ```js import { test as base } from '@playwright/test'; const test = base.extend<{ testFixture: string, autoTestFixture: string, unusedFixture: string, }, { workerFixture: string, autoWorkerFixture: string, }>({ workerFixture: [async ({ browser }) => { // workerFixture setup... await use('workerFixture'); // workerFixture teardown... }, { scope: 'worker' }], autoWorkerFixture: [async ({ browser }) => { // autoWorkerFixture setup... await use('autoWorkerFixture'); // autoWorkerFixture teardown... }, { scope: 'worker', auto: true

test_fixtures#with-fixturesrank 2 · score 0.714not in golden set

Fixtures have a number of advantages over before/after hooks: - Fixtures **encapsulate** setup and teardown in the same place so it is easier to write. So if you have an after hook that tears down what was created in a before hook, consider turning them into a fixture. - Fixtures are **reusable** between test files - you can define them once and use them in all your tests. That's how Playwright's built-in `page` fixture works. So if you have a helper function that is used in multiple tests, consider turning it into a fixture. - Fixtures are **on-demand** - you can define as many fixtures as you'd like, and Playwright Test will setup only the ones needed by your test and nothing else. - Fixtures are **composable** - they can depend on each other to provide complex behaviors. - Fixtures are **flexible**. Tests can use any combination of fixtures to precisely tailor the environment to their needs, without affecting other tests. - Fixtures simplify **grouping**. You no longer need to wrap tests in `describe`s that set up their environment, and

test_fixtures#fixture-timeoutrank 3 · score 0.677not in golden set

Fixture is considered to be a part of a test, and so its setup and teardown running time counts towards the test timeout. Therefore, a slow fixture may cause test timeouts. You can set a separate larger timeout for such a fixture, and keep the overall test timeout small. ```js import { test as base, expect } from '@playwright/test'; const test = base.extend<{ slowFixture: string }>({ slowFixture: [async ({}, use) => { // ... perform a slow operation ... await use('hello'); }, { timeout: 60000 }] }); test('example test', async ({ slowFixture }) => { // ... }); ``` Unlike regular test-scoped fixtures, each [worker-scoped](#worker-scoped-fixtures) fixture has its own timeout, equal to the test timeout. You can change the timeout for a worker-scoped fixture in the same way.

Faithfulness

Heuristic grounding score 0.64 over 1 claim(s).

failcfg-project-dependenciesHow do I make a login setup step run before my chromium and firefox projects?0.50

Answer

You can create a new `setup` project in your config and declare it as a dependency for your chromium and firefox testing projects [auth#basic-shared-account-in-all-tests~2]. Additionally, configure the setup project with a `testMatch` pattern (such as `/.*\.setup\.ts/`) so that it always runs before all the tests [auth#basic-shared-account-in-all-tests~2, auth#basic-shared-account-in-all-tests~3].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
fail	keyword_presence	0.00	missing: dependencies
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 2 retrieved chunk(s)
fail	expected_citation_overlap	0.00	cited 0/1 expected chunk(s)
fail	grounding	0.50	1 unsupported claim(s): Additionally, configure the setup project with a `testMatch` pattern (such as `/.*\.setup\
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	0.000
precision@3	0.000
recall@3	0.000
reciprocal_rank	0.000

Retrieved chunks

auth#basic-shared-account-in-all-tests~3rank 1 · score 0.797not in golden set

testMatch: /.*\.setup\.ts/ }, { name: 'chromium', use: { ...devices['Desktop Chrome'], // Use prepared auth state. storageState: 'playwright/.auth/user.json', }, dependencies: ['setup'], }, { name: 'firefox', use: { ...devices['Desktop Firefox'], // Use prepared auth state. storageState: 'playwright/.auth/user.json', }, dependencies: ['setup'], }, ], }); ``` Tests start already authenticated because we specified `storageState` in the config. ```js import { test } from '@playwright/test'; test('test', async ({ page }) => { // page is authenticated }); ``` Note that you need to delete the stored state when it expires. If you don't need to keep the state between test runs, write the browser state under testProject.outputDir, which is automatically cleaned up before every test run.

auth#basic-shared-account-in-all-tests~2rank 2 · score 0.767not in golden set

own. await page.goto('https://github.com/login'); await page.getByLabel('Username or email address').fill('username'); await page.getByLabel('Password').fill('password'); await page.getByRole('button', { name: 'Sign in' }).click(); // Wait until the page receives the cookies. // // Sometimes login flow sets cookies in the process of several redirects. // Wait for the final URL to ensure that the cookies are actually set. await page.waitForURL('https://github.com/'); // Alternatively, you can wait until the page reaches a state where all cookies are set. await expect(page.getByRole('button', { name: 'View profile and more' })).toBeVisible(); // End of authentication steps. await page.context().storageState({ path: authFile }); }); ``` Create a new `setup` project in the config and declare it as a [dependency](./test-projects.md#dependencies) for all your testing projects. This project will always run and authenticate before all the tests. All testing projects should use the authenticated state as `storageState`. ```js import { defineConfig, devices } from '@playwright/test'; export default defineConfig({ projects: [ // Setup project { name: 'setup', testMatch: /.*\.setup\.ts/ }, { name: 'chromium', use: { ...devices['Desktop Chrome'], // Use prepared auth state. storageState: 'playwright/.auth/user.json', }, dependencies: ['setup'], }, { name: 'firefox', use: { ...devices['Desktop Firefox'], // Use

test_parallel#parallelize-tests-in-a-single-file~2rank 3 · score 0.742not in golden set

export default defineConfig({ // runs all tests in all files of a specific project in parallel projects: [ { name: 'chromium', use: { ...devices['Desktop Chrome'] }, fullyParallel: true, }, ] }); ```

Expected but not retrieved: test_projects#dependencies

Faithfulness

Heuristic grounding score 0.50 over 2 claim(s).

Ungrounded claim(s):

Additionally, configure the setup project with a `testMatch` pattern (such as `/.*\.setup\.ts/`) so that it always runs before all the tests [auth#basic-shared-account-in-all-tests~2, auth#basic-shared-account-in-all-tests~3].

passcfg-run-one-projectHow do I run only the firefox project from the command line?0.73

Answer

To run tests on the Firefox project, use the `--project` flag followed by `firefox` [running_tests#run-tests-on-different-browsers].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
fail	expected_citation_overlap	0.00	cited 0/1 expected chunk(s)
pass	grounding	0.68	all 1 claim(s) grounded (score 0.68)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	0.000
precision@3	0.000
recall@3	0.000
reciprocal_rank	0.000

Retrieved chunks

running_tests#run-tests-on-different-browsersrank 1 · score 0.740not in golden set

To specify which browser you would like to run your tests on, use the `--project` flag followed by the browser name. ```bash npx playwright test --project webkit ``` To specify multiple browsers to run your tests on, use the `--project` flag multiple times followed by each browser name. ```bash npx playwright test --project webkit --project firefox ```

browsers#run-tests-on-different-browsers-3rank 2 · score 0.724not in golden set

Run tests on a specific browser: Run tests on multiple browsers and make it based on the environment variable `BROWSER`:

debug#debug-one-test-on-a-specific-browserrank 3 · score 0.719not in golden set

To run one test on a specific browser add the name of the test file and the line number of the test you want to debug as well as the `--project` flag followed by the name of the project. ```bash npx playwright test example.spec.ts:10 --project=webkit --debug ```

Expected but not retrieved: test_projects#run-projects

Faithfulness

Heuristic grounding score 0.68 over 1 claim(s).

failcfg-default-reporter-ciWhich reporter is used by default on CI?0.68

Answer

The `dot` reporter is used by default on CI [test_cli#options-4, test_reporters#reporters-on-ci].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 2 retrieved chunk(s)
fail	expected_citation_overlap	0.00	cited 0/2 expected chunk(s)
fail	grounding	0.40	1 unsupported claim(s): The `dot` reporter is used by default on CI [test_cli#options-4, test_reporters#reporters-
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	0.500
reciprocal_rank	0.333

Retrieved chunks

test_cli#options-4rank 1 · score 0.792not in golden set

test_reporters#reporters-on-cirank 2 · score 0.779not in golden set

You can use different reporters locally and on CI. For example, using concise `'dot'` reporter avoids too much output. This is the default on CI. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Concise 'dot' for CI, default 'list' when running locally reporter: process.env.CI ? 'dot' : 'list', }); ```

test_reporters#list-reporterrank 3 · score 0.727expected

List reporter is default (except on CI where the `dot` reporter is default). It prints a line for each test being run. ```bash npx playwright test --reporter=list ``` ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ reporter: 'list', }); ``` Here is an example output in the middle of a test run. Failures will be listed at the end by default. ```bash npx playwright test --reporter=list Running 124 tests using 6 workers 1 ✓ should access error in env (438ms) 2 ✓ handle long test names (515ms) 3 x 1) render expected (691ms) 4 ✓ should timeout (932ms) 5 should repeat each: 6 ✓ should respect enclosing .gitignore (569ms) 7 should teardown env after timeout: 8 should respect excluded tests: 9 ✓ should handle env beforeEach error (638ms) 10 should respect enclosing .gitignore: ``` You can opt into the step rendering via passing the following config option: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ reporter: [['list', { printSteps: true }]], }); ``` You can print failures inline as soon as they are available instead of

Expected but not retrieved: test_reporters#dot-reporter

Faithfulness

Heuristic grounding score 0.40 over 1 claim(s).

Ungrounded claim(s):

The `dot` reporter is used by default on CI [test_cli#options-4, test_reporters#reporters-on-ci].

failcfg-html-report-openHow do I stop the HTML report from opening in a browser automatically after a failed run?0.90

Answer

You can stop the HTML report from opening automatically by setting the `open` property to `'never'` in the Playwright config [test_reporters#html-reporter].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
fail	grounding	0.49	1 unsupported claim(s): You can stop the HTML report from opening automatically by setting the `open` property to
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	0.333

Retrieved chunks

intro#html-test-reportsrank 1 · score 0.819not in golden set

After a test run, the [HTML Reporter](./test-reporters.md#html-reporter) provides a dashboard filterable by the browser, passed, failed, skipped, flaky and more. Click a test to inspect errors, attachments and steps. It auto-opens only when failures occur; open manually with the command below. <Tabs groupId="js-package-manager" defaultValue="npm" values={[ {label: 'npm', value: 'npm'}, {label: 'yarn', value: 'yarn'}, {label: 'pnpm', value: 'pnpm'} ] }> <TabItem value="npm"> ```bash npx playwright show-report ``` </TabItem> <TabItem value="yarn"> ```bash yarn playwright show-report ``` </TabItem> <TabItem value="pnpm"> ```bash pnpm exec playwright show-report ``` </TabItem> </Tabs> ![HTML Report](./images/getting-started/html-report-basic.png)

running_tests#test-reportsrank 2 · score 0.794not in golden set

The [HTML Reporter](./test-reporters.md#html-reporter) shows you a full report of your tests allowing you to filter the report by browsers, passed tests, failed tests, skipped tests, and flaky tests. By default, the HTML report opens automatically if some tests failed, otherwise you can open it with the following command. ```bash npx playwright show-report ``` ![HTML Report](./images/getting-started/html-report.png) You can filter and search for tests as well as click on each test to see the test errors and explore each step of the test. ![HTML Reporter detail view](./images/getting-started/html-report-detail.png)

test_reporters#html-reporterrank 3 · score 0.764expected

HTML reporter produces a self-contained folder that contains report for the test run that can be served as a web page. ```bash npx playwright test --reporter=html ``` By default, HTML report is opened automatically if some of the tests failed. You can control this behavior via the `open` property in the Playwright config or the `PLAYWRIGHT_HTML_OPEN` environmental variable. The possible values for that property are `always`, `never` and `on-failure` (default). You can also configure `host` and `port` that are used to serve the HTML report. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ reporter: [['html', { open: 'never' }]], }); ``` By default, report is written into the `playwright-report` folder in the current working directory. One can override that location using the `PLAYWRIGHT_HTML_OUTPUT_DIR` environment variable or a reporter configuration. In configuration file, pass options directly: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ reporter: [['html', { outputFolder: 'my-report' }]], }); ``` If you are uploading attachments from a data folder to another location, you can use `attachmentsBaseURL` option to let html report know where to look

Faithfulness

Heuristic grounding score 0.49 over 1 claim(s).

Ungrounded claim(s):

You can stop the HTML report from opening automatically by setting the `open` property to `'never'` in the Playwright config [test_reporters#html-reporter].

passcfg-github-annotationsIs there a reporter that annotates failures directly in GitHub Actions?0.99

Answer

Yes, you can use the built-in `github` reporter to get automatic failure annotations when running in GitHub actions [test_reporters#github-actions-annotations].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.93	all 1 claim(s) grounded (score 0.93)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

test_reporters#github-actions-annotationsrank 1 · score 0.871expected

You can use the built in `github` reporter to get automatic failure annotations when running in GitHub actions. Note that all other reporters work on GitHub Actions as well, but do not provide annotations. Also, it is not recommended to use this annotation type if running your tests with a matrix strategy as the stack trace failures will multiply and obscure the GitHub file view. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // 'github' for GitHub Actions CI to generate annotations, plus a concise 'dot' // default 'list' when running locally reporter: process.env.CI ? 'github' : 'list', }); ``` The `github` reporter accepts `omitTags` (or the `PLAYWRIGHT_GITHUB_OMIT_TAGS` environment variable) to suppress test tags in its annotations, for example `reporter: [['github', { omitTags: true }]]`.

ci_intro#opening-the-workflowsrank 2 · score 0.759not in golden set

Click on the **Actions** tab to see the workflows. Here you see if your tests have passed or failed. ###### <img src="https://user-images.githubusercontent.com/13063165/183423783-58bf2008-514e-4f96-9c12-c9a55703960c.png" alt="opening the workflow" width="1678" height="1207" /> ###### <img src="https://github.com/microsoft/playwright/assets/13063165/71793c09-0815-4faa-866b-85684a1f87e5" alt="opening the workflow" width="2812" height="824" /> On Pull Requests you can also click on the **Details** link in the [PR status check](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/collaborating-on-repositories-with-code-quality-features/about-status-checks). <img height="404" width="1290" alt="pr status checked" src="https://user-images.githubusercontent.com/13063165/183722462-17a985db-0e10-4205-b16c-8aaac36117b9.png" />

ci_intro#viewing-test-logsrank 3 · score 0.755not in golden set

Clicking on the workflow run shows you all the actions that GitHub performed and clicking on **Run Playwright tests** shows the error messages, what was expected and what was received as well as the call log. ###### <img src="https://user-images.githubusercontent.com/13063165/183423783-58bf2008-514e-4f96-9c12-c9a55703960c.png" alt="Viewing Test Logs" width="1678" height="1207" /> ###### <img src="https://github.com/microsoft/playwright/assets/13063165/ba2d8d7b-ffce-42de-95e0-bcb35c421975" alt="viewing the test logs" width="2794" height="1858" />

Faithfulness

Heuristic grounding score 0.93 over 1 claim(s).

passcfg-webserverCan Playwright start my app's dev server before the tests instead of me running it separately?0.84

Answer

Yes, Playwright includes a `webServer` option in its configuration file to launch a local dev server before running your tests [test_webserver#introduction].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	0.50	cited 1/2 expected chunk(s)
pass	grounding	0.72	all 1 claim(s) grounded (score 0.72)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.667
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

test_webserver#introductionrank 1 · score 0.871expected

Playwright comes with a `webServer` option in the config file which gives you the ability to launch a local dev server before running your tests. This is ideal for when writing your tests during development and when you don't have a staging or production url to test against.

test_webserver#configuring-a-web-serverrank 2 · score 0.847expected

Use the `webServer` property in your Playwright config to launch a development web server during the tests. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Run your local dev server before starting the tests webServer: { command: 'npm run start', url: 'http://localhost:3000', reuseExistingServer: !process.env.CI, stdout: 'ignore', stderr: 'pipe', }, }); ``` | Property | Description | | :- | :- | | testConfig.webServer | Launch a development web server (or multiple) during the tests. | | `command`| Shell command to start the local dev server of your app. | | `cwd` | Current working directory of the spawned process, defaults to the directory of the configuration file. | | `env` | Environment variables for the command. Defaults to inheriting `process.env` with `PLAYWRIGHT_TEST=1` added. | | `gracefulShutdown` | How to shut down the process. If unspecified, the process group is forcefully `SIGKILL`ed. If set to `{ signal: 'SIGTERM', timeout: 500 }`, the process group is sent a `SIGTERM` signal, followed by `SIGKILL` if it doesn't exit within 500ms. You can also use `SIGINT` as the signal instead. A

test_webserver#adding-a-server-timeoutrank 3 · score 0.826not in golden set

Webservers can sometimes take longer to boot up. In this case, you can increase the timeout to wait for the server to start. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Rest of your config... // Run your local dev server before starting the tests webServer: { command: 'npm run start', url: 'http://localhost:3000', reuseExistingServer: !process.env.CI, timeout: 120 * 1000, }, }); ```

Faithfulness

Heuristic grounding score 0.72 over 1 claim(s).

passcfg-update-snapshotsThe UI changed on purpose and my screenshot comparisons fail. How do I regenerate the baselines?0.94

Answer

You can update the reference screenshot by using the `--update-snapshots` flag [test_snapshots#updating-screenshots].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.72	all 1 claim(s) grounded (score 0.72)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

test_snapshots#updating-screenshotsrank 1 · score 0.749expected

Sometimes you need to update the reference screenshot, for example when the page has changed. Do this with the `--update-snapshots` flag. ```bash npx playwright test --update-snapshots ```

test_ui_mode#attachmentsrank 2 · score 0.719not in golden set

The "Attachments" tab allows you to explore attachments. If you're doing [visual regression testing](./test-snapshots.md), you'll be able to compare screenshots by examining the image diff, the actual image and the expected image. When you click on the expected image you can use the slider to slide one image over the other so you can easily see the differences in your screenshots. <img src="https://github.com/microsoft/playwright/assets/13063165/bb83b406-84ed-4380-a96c-0e62d1388093" alt="ui mode with attachments" width="3606" height="2228" />

trace_viewer#attachmentsrank 3 · score 0.715not in golden set

The "Attachments" tab allows you to explore attachments. If you're doing [visual regression testing](./test-snapshots.md), you'll be able to compare screenshots by examining the image diff, the actual image and the expected image. When you click on the expected image you can use the slider to slide one image over the other so you can easily see the differences in your screenshots. <img src="https://github.com/microsoft/playwright/assets/13063165/4386178a-5808-4fa8-9436-315350a23b04" alt="attachments tab in trace viewer" width="3598" height="2218" />

Faithfulness

Heuristic grounding score 0.72 over 1 claim(s).

failcfg-max-diff-pixelsHow do I allow a small number of differing pixels in toHaveScreenshot?0.85

Answer

You can allow a small number of differing pixels by using the `maxDiffPixels` option in `toHaveScreenshot` or by specifying it globally in your Playwright config [test_snapshots#maxdiffpixels].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
fail	grounding	0.26	1 unsupported claim(s): You can allow a small number of differing pixels by using the `maxDiffPixels` option in `t
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	0.333

Retrieved chunks

test_snapshots#stylepathrank 1 · score 0.729not in golden set

You can apply a custom stylesheet to your page while taking screenshot. This allows filtering out dynamic or volatile elements, hence improving the screenshot determinism. ```css iframe { visibility: hidden; } ``` ```js import { test, expect } from '@playwright/test'; test('example test', async ({ page }) => { await page.goto('https://playwright.dev'); await expect(page).toHaveScreenshot({ stylePath: path.join(__dirname, 'screenshot.css') }); }); ``` If you'd like to share the default value among all the tests in the project, you can specify it in the playwright config, either globally or per project: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ expect: { toHaveScreenshot: { stylePath: './screenshot.css' }, }, }); ```

screenshots#capture-into-bufferrank 2 · score 0.726not in golden set

Rather than writing into a file, you can get a buffer with the image and post-process it or pass it to a third party pixel diff facility. ```js const buffer = await page.screenshot(); console.log(buffer.toString('base64')); ```

test_snapshots#maxdiffpixelsrank 3 · score 0.723expected

Playwright Test uses the [pixelmatch](https://github.com/mapbox/pixelmatch) library. You can [pass various options](./api/class-pageassertions.md#page-assertions-to-have-screenshot-1) to modify its behavior: ```js import { test, expect } from '@playwright/test'; test('example test', async ({ page }) => { await page.goto('https://playwright.dev'); await expect(page).toHaveScreenshot({ maxDiffPixels: 100 }); }); ``` If you'd like to share the default value among all the tests in the project, you can specify it in the playwright config, either globally or per project: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ expect: { toHaveScreenshot: { maxDiffPixels: 100 }, }, }); ```

Faithfulness

Heuristic grounding score 0.26 over 1 claim(s).

Ungrounded claim(s):

You can allow a small number of differing pixels by using the `maxDiffPixels` option in `toHaveScreenshot` or by specifying it globally in your Playwright config [test_snapshots#maxdiffpixels].

failcfg-video-retainHow do I record video only for tests that fail?0.91

Answer

To record video only for tests that fail, you can use the `'retain-on-failure'` option for the `video` setting in your Playwright configuration [videos#record-video].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
fail	grounding	0.54	grounding score 0.54 below threshold 0.55
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

videos#record-videorank 1 · score 0.817expected

Playwright Test can record videos for your tests, controlled by the `video` option in your Playwright config. By default videos are off. - `'off'` - Do not record video. - `'on'` - Record video for each test. - `'retain-on-failure'` - Record video for each test, but remove all videos from successful test runs. - `'on-first-retry'` - Record video only when retrying a test for the first time. Video files will appear in the test output directory, typically `test-results`. See testOptions.video for advanced video configuration. Videos are saved upon [browser context](./browser-contexts.md) closure at the end of a test. If you create a browser context manually, make sure to await browserContext.close(). ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ use: { video: 'on-first-retry', }, }); ``` ```js const context = await browser.newContext({ recordVideo: { dir: 'videos/' } }); // Make sure to await close, so that videos are saved. await context.close(); ``` You can also specify video size and annotation. The video size defaults to the viewport size scaled down to fit 800x800. The video of the viewport is placed

test_use_options#recording-optionsrank 2 · score 0.775not in golden set

With Playwright you can capture screenshots, record videos as well as traces of your test. By default these are turned off but you can enable them by setting the `screenshot`, `video` and `trace` options in your `playwright.config.js` file. Trace files, screenshots and videos will appear in the test output directory, typically `test-results`. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ use: { // Capture screenshot after each test failure. screenshot: 'only-on-failure', // Record trace only when retrying a test for the first time. trace: 'on-first-retry', // Record video only when retrying a test for the first time. video: 'on-first-retry' }, }); ``` | Option | Description | | :- | :- | | testOptions.screenshot | Capture [screenshots](./screenshots.md) of your test. Options include `'off'`, `'on'` and `'only-on-failure'` | | testOptions.trace | Playwright can produce test traces while running the tests. Later on, you can view the trace and get detailed information about Playwright execution by opening [Trace Viewer](./trace-viewer.md). Options include: `'off'`, `'on'`, `'retain-on-failure'` and `'on-first-retry'` | | testOptions.video | Playwright can record [videos](./videos.md) for your tests. Options include: `'off'`,

trace_viewer#run-trace-only-on-failurerank 3 · score 0.775not in golden set

Setup your tests to record a trace only when the test fails: <Tabs groupId="test-runners" defaultValue="mstest" values={[ {label: 'MSTest', value: 'mstest'}, {label: 'NUnit', value: 'nunit'}, {label: 'xUnit', value: 'xunit'}, {label: 'xUnit v3', value: 'xunit-v3'}, ] }> <TabItem value="nunit"> </TabItem> <TabItem value="mstest"> </TabItem> <TabItem value="xunit"> </TabItem> <TabItem value="xunit-v3"> </TabItem> </Tabs>

Faithfulness

Heuristic grounding score 0.54 over 1 claim(s).

failcfg-tag-testsHow do I tag tests as smoke so I can run just those?0.00

Answer

I do not have enough information in the provided documents to answer that.

Checks

	Check	Detail
skip	exact_match	no expected_answer in dataset
skip	fuzzy_match	no expected_answer in dataset
fail	keyword_presence	missing: @
skip	forbidden_absent	no forbidden_keywords in dataset
fail	refusal_behaviour	refused a question it should have answered
skip	citation_present	refusals are not expected to cite sources
fail	expected_citation_overlap	cited 0/1 expected chunk(s)
skip	grounding	skipped: refusals are not grounded in retrieved text by design
skip	llm_judge_faithfulness	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	0.500

Retrieved chunks

test_projects#splitting-tests-into-projectsrank 1 · score 0.774not in golden set

We can split tests into projects and use filters to run a subset of tests. For example, we can create a project that runs tests using a filter matching all tests with a specific file name. We can then have another group of tests that ignore specific test files. Here is an example that defines a common timeout and two projects. The "Smoke" project runs a small subset of tests without retries, and "Default" project runs all other tests with retries. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ timeout: 60000, // Timeout is shared between all tests. projects: [ { name: 'Smoke', testMatch: /.*smoke.spec.ts/, retries: 0, }, { name: 'Default', testIgnore: /.*smoke.spec.ts/, retries: 2, }, ], }); ```

test_annotations#tag-testsrank 2 · score 0.767expected

Sometimes you want to tag your tests as `@fast` or `@slow`, and then filter by tag in the test report. Or you might want to only run tests that have a certain tag. To tag a test, either provide an additional details object when declaring a test, or add `@`-token to the test title. Note that tags must start with `@` symbol. ```js import { test, expect } from '@playwright/test'; test('test login page', { tag: '@fast', }, async ({ page }) => { // ... }); test('test full report @slow', async ({ page }) => { // ... }); ``` You can also tag all tests in a group or provide multiple tags: ```js import { test, expect } from '@playwright/test'; test.describe('group', { tag: '@report', }, () => { test('test report header', async ({ page }) => { // ... }); test('test full report', { tag: ['@slow', '@vrt'], }, async ({ page }) => { // ... }); }); ``` You can now run tests that have a particular tag with [`--grep`](./test-cli.md#all-options) command line option. ```bash npx playwright test --grep

test_sharding#merging-reports-from-multiple-environmentsrank 3 · score 0.760not in golden set

If you want to run the same tests in multiple environments, as opposed to shard your tests onto multiple machines, you need to differentiate these environments. In this case, it is useful to specify the testConfig.tag property, to tag all tests with the environment name. This tag will be automatically picked up by the blob report and later on by the merge tool. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ reporter: process.env.CI ? 'blob' : 'html', tag: process.env.CI_ENVIRONMENT_NAME, // for example "@APIv2" }); ```

Faithfulness

Heuristic grounding score 0.00 over 1 claim(s).

Ungrounded claim(s):

I do not have enough information in the provided documents to answer that.

failcfg-focus-testHow do I run only one test while I'm debugging it?0.56

Answer

To debug one test, run the test command followed by the test file name, the line number, and the `--debug` flag [debug#debug-one-test-on-all-browsers]. For example, use `npx playwright test example.spec.ts:10 --debug` [debug#debug-one-test-on-all-browsers].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
fail	keyword_presence	0.00	missing: test.only
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
fail	expected_citation_overlap	0.00	cited 0/1 expected chunk(s)
pass	grounding	0.79	all 2 claim(s) grounded (score 0.79)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	0.000
precision@3	0.000
recall@3	0.000
reciprocal_rank	0.000

Retrieved chunks

debug#debug-one-test-on-all-browsersrank 1 · score 0.790not in golden set

To debug one test on a specific line, run the test command followed by the name of the test file and the line number of the test you want to debug, followed by the `--debug` flag. This will run a single test in each browser configured in your [`playwright.config`](./test-projects.md#configure-projects-for-multiple-browsers) and open the inspector. ```bash npx playwright test example.spec.ts:10 --debug ```

running_tests#introductionrank 2 · score 0.788not in golden set

With Playwright you can run a single test, a set of tests, or all tests. Tests can be run on one browser or multiple browsers using the `--project` flag. Tests run in parallel by default and in headless mode, meaning no browser window opens while running the tests and results appear in the terminal. You can run tests in headed mode using the `--headed` CLI argument, or run your tests in [UI mode](./test-ui-mode.md) using the `--ui` flag to see a full trace of your tests. **You will learn** - [How to run tests from the command line](/running-tests.md#command-line) - [How to debug tests](/running-tests.md#debugging-tests) - [How to open the HTML test reporter](/running-tests.md#test-reports)

test_global_setup_teardown#test-filteringrank 3 · score 0.780not in golden set

All test filtering options, such as `--grep`/`--grep-invert`, `--shard`, filtering directly by location in the command line, or using [`test.only()`](./api/class-test.md#test-only), directly select the primary tests to be run. If those tests belong to a project with dependencies, all tests from those dependencies will also run. You can pass `--no-deps` command line option to ignore all dependencies and teardowns. Only your directly selected projects will run.

Expected but not retrieved: test_annotations#focus-a-test

Faithfulness

Heuristic grounding score 0.79 over 2 claim(s).

failnet-mock-apiHow do I return fake JSON for an API call so the test never hits the real backend?0.79

Answer

To return fake JSON for an API call without hitting the real backend, you can use `page.route` to intercept the call and fulfill it with custom JSON data [mock#mock-api-requests]. For example, call `await route.fulfill({ json })` within the route handler before navigating to the page [mock#mock-api-requests].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	0.50	cited 1/2 expected chunk(s)
fail	grounding	0.47	1 unsupported claim(s): To return fake JSON for an API call without hitting the real backend, you can use `page.ro
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.667
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

mock#mock-api-requestsrank 1 · score 0.815expected

The following code will intercept all the calls to `*/**/api/v1/fruits` and will return a custom response instead. No requests to the API will be made. The test goes to the URL that uses the mocked route and asserts that mock data is present on the page. ```js test("mocks a fruit and doesn't call api", async ({ page }) => { // Mock the api call before navigating await page.route('*/**/api/v1/fruits', async route => { const json = [{ name: 'Strawberry', id: 21 }]; await route.fulfill({ json }); }); // Go to the page await page.goto('https://demo.playwright.dev/api-mocking'); // Assert that the Strawberry fruit is visible await expect(page.getByText('Strawberry')).toBeVisible(); }); ``` You can see from the trace of the example test that the API was never called, it was however fulfilled with the mock data. <img src="https://github.com/microsoft/playwright/assets/13063165/3dc14cbf-c100-4efc-ac21-d7b52d698b53" alt="api mocking trace" width="2946" height="1902" /> Read more about [advanced networking](./network.md).

mock#modify-api-responsesrank 2 · score 0.795not in golden set

Sometimes, it is essential to make an API request, but the response needs to be patched to allow for reproducible testing. In that case, instead of mocking the request, one can perform the request and fulfill it with the modified response. In the example below we intercept the call to the fruit API and add a new fruit called 'Loquat', to the data. We then go to the url and assert that this data is there: ```js test('gets the json from api and adds a new fruit', async ({ page }) => { // Get the response and add to it await page.route('*/**/api/v1/fruits', async route => { const response = await route.fetch(); const json = await response.json(); json.push({ name: 'Loquat', id: 100 }); // Fulfill using the original response, while patching the response body // with the given JSON object. await route.fulfill({ response, json }); }); // Go to the page await page.goto('https://demo.playwright.dev/api-mocking'); // Assert that the new fruit is visible await expect(page.getByText('Loquat', { exact: true })).toBeVisible(); }); ``` In the trace of our test we can see that the

network#handle-requestsrank 3 · score 0.763expected

```js await page.route('**/api/fetch_data', route => route.fulfill({ status: 200, body: testData, })); await page.goto('https://example.com'); ``` You can mock API endpoints via handling the network requests in your Playwright script.

Faithfulness

Heuristic grounding score 0.47 over 2 claim(s).

Ungrounded claim(s):

To return fake JSON for an API call without hitting the real backend, you can use `page.route` to intercept the call and fulfill it with custom JSON data [mock#mock-api-requests].

passnet-abort-imagesCan I block all image requests to make tests faster?0.92

Answer

Yes, you can abort requests using `page.route()` and `route.abort()` to block image requests like png and jpeg [network#abort-requests, network#network-mocking].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 2 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.61	all 1 claim(s) grounded (score 0.61)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

network#abort-requestsrank 1 · score 0.789expected

You can abort requests using page.route() and route.abort(). ```js await page.route('**/*.{png,jpg,jpeg}', route => route.abort()); // Abort based on the request type await page.route('**/*', route => { return route.request().resourceType() === 'image' ? route.abort() : route.continue(); }); ```

network#network-mockingrank 2 · score 0.736not in golden set

You don't have to configure anything to mock network requests. Just define a custom Route that mocks network for a browser context. ```js import { test, expect } from '@playwright/test'; test.beforeEach(async ({ context }) => { // Block any css requests for each test in this file. await context.route(/.css$/, route => route.abort()); }); test('loads page without css', async ({ page }) => { await page.goto('https://playwright.dev'); // ... test goes here }); ``` Alternatively, you can use page.route() to mock network in a single page. ```js import { test, expect } from '@playwright/test'; test('loads page without images', async ({ page }) => { // Block png and jpeg images. await page.route(/(png|jpeg)$/, route => route.abort()); await page.goto('https://playwright.dev'); // ... test goes here }); ```

debug#run-a-test-from-a-specific-breakpointrank 3 · score 0.712not in golden set

To speed up the debugging process you can add a page.pause() method to your test. This way you won't have to step through each action of your test to get to the point where you want to debug. ```js await page.pause(); ``` Once you add a `page.pause()` call, run your tests in debug mode. Clicking the "Resume" button in the Inspector will run the test and only stop on the `page.pause()`. <img height="1736" width="2700" alt="test with page.pause" src="https://user-images.githubusercontent.com/13063165/219473050-122be4c2-31d0-4cbd-aa8b-8588e8b781a6.png" />

Faithfulness

Heuristic grounding score 0.61 over 1 claim(s).

failnet-glob-double-starIn a page.route pattern, what is the difference between * and **?0.00

Answer

I do not have enough information in the provided documents to answer that.

Checks

	Check	Detail
skip	exact_match	no expected_answer in dataset
skip	fuzzy_match	no expected_answer in dataset
fail	keyword_presence	missing: except, /
skip	forbidden_absent	no forbidden_keywords in dataset
fail	refusal_behaviour	refused a question it should have answered
skip	citation_present	refusals are not expected to cite sources
fail	expected_citation_overlap	cited 0/1 expected chunk(s)
skip	grounding	skipped: refusals are not grounded in retrieved text by design
skip	llm_judge_faithfulness	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	0.000
precision@3	0.000
recall@3	0.000
reciprocal_rank	0.000

Retrieved chunks

evaluating#evaluation-argumentrank 1 · score 0.720not in golden set

Playwright evaluation methods like page.evaluate() take a single optional argument. This argument can be a mix of Serializable values and JSHandle instances. Handles are automatically converted to the value they represent. ```js // A primitive value. await page.evaluate(num => num, 42); // An array. await page.evaluate(array => array.length, [1, 2, 3]); // An object. await page.evaluate(object => object.foo, { foo: 'bar' }); // A single handle. const button = await page.evaluateHandle('window.button'); await page.evaluate(button => button.textContent, button); // Alternative notation using JSHandle.evaluate. await button.evaluate((button, from) => button.textContent.substring(from), 5); // Object with multiple handles. const button1 = await page.evaluateHandle('window.button1'); const button2 = await page.evaluateHandle('window.button2'); await page.evaluate( o => o.button1.textContent + o.button2.textContent, { button1, button2 }); // Object destructuring works. Note that property names must match // between the destructured object and the argument. // Also note the required parenthesis. await page.evaluate( ({ button1, button2 }) => button1.textContent + button2.textContent, { button1, button2 }); // Array works as well. Arbitrary names can be used for destructuring. // Note the required parenthesis. await page.evaluate( ([b1, b2]) => b1.textContent + b2.textContent, [button1, button2]); // Any

pages#pagesrank 2 · score 0.710not in golden set

Each BrowserContext can have multiple pages. A Page refers to a single tab or a popup window within a browser context. It should be used to navigate to URLs and interact with the page content. ```js // Create a page. const page = await context.newPage(); // Navigate explicitly, similar to entering a URL in the browser. await page.goto('http://example.com'); // Fill an input. await page.locator('#search').fill('query'); // Navigate implicitly by clicking a link. await page.locator('#submit').click(); // Expect a new url. console.log(page.url()); ```

other_locators#xpath-locatorrank 3 · score 0.707not in golden set

We recommend prioritizing [user-visible locators](./locators.md#quick-guide) like text or accessible role instead of using XPath that is tied to the implementation and easily break when the page changes. XPath locators are equivalent to calling [`Document.evaluate`](https://developer.mozilla.org/en/docs/Web/API/Document/evaluate). ```js await page.locator('xpath=//button').click(); ``` Any selector string starting with `//` or `..` are assumed to be an xpath selector. For example, Playwright converts `'//html/body'` to `'xpath=//html/body'`. XPath does not pierce shadow roots.

Expected but not retrieved: network#glob-url-patterns

Faithfulness

Heuristic grounding score 0.04 over 1 claim(s).

Ungrounded claim(s):

I do not have enough information in the provided documents to answer that.

passnet-redirect-handlerIf a request redirects, does my route handler get called again for the redirected request?0.81

Answer

No, the handler is only called once for the original request, and the browser follows the redirect on its own [network#redirects].

Checks

	Check	Score	Detail
fail	exact_match	0.00	answer differs from expected_answer
pass	fuzzy_match	0.90	similarity 0.90 vs threshold 0.60
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.77	all 1 claim(s) grounded (score 0.77)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

network#redirectsrank 1 · score 0.809expected

Playwright treats a request and its redirects as a single unit. The handler is called once, for the original request, and the browser follows the redirect on its own. response.request() returns the last request in the chain, and request.redirectedFrom() walks it back to the one you intercepted. Headers passed to route.continue() apply to every hop of the chain, except `cookie`, which always comes from the cookie store. Fulfilling with a `3xx` status does not give you a second chance to intercept. Chromium and Firefox follow the redirect without calling your handler, and WebKit rejects the call. To serve different content, fulfill with that content. To send the request elsewhere, pass `url` to route.continue() or route.fallback().

network#how-request-interception-worksrank 2 · score 0.767not in golden set

Routes sit between the page and the browser's network stack. The handler runs before the network stack has processed the request: route.continue() passes it on, route.fulfill() answers it without touching the network, and route.abort() fails it.

network#network-eventsrank 3 · score 0.727not in golden set

You can monitor all the Requests and Responses: ```js // Subscribe to 'request' and 'response' events. page.on('request', request => console.log('>>', request.method(), request.url())); page.on('response', response => console.log('<<', response.status(), response.url())); await page.goto('https://example.com'); ``` Or wait for a network response after the button click with page.waitForResponse(): ```js // Use a glob URL pattern. Note no await. const responsePromise = page.waitForResponse('**/api/fetch_data'); await page.getByText('Update').click(); const response = await responsePromise; ```

Faithfulness

Heuristic grounding score 0.77 over 1 claim(s).

passnet-msw-missing-eventsI use Mock Service Worker in the app and page.route never sees the requests. What is going on?0.97

Answer

Mock Service Worker adds its own Service Worker that takes over network requests, making them invisible to Playwright's `page.route()` and `browserContext.route()` [network#missing-network-events-and-service-workers]. To fix this, you can disable Service Workers by setting `browser.newContext.serviceWorkers` to `'block'` [network#missing-network-events-and-service-workers].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.83	all 2 claim(s) grounded (score 0.83)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

network#missing-network-events-and-service-workersrank 1 · score 0.805expected

Playwright's built-in browserContext.route() and page.route() allow your tests to natively route requests and perform mocking and interception. If you're using Playwright's native browserContext.route() and page.route(), and it appears network events are missing, disable Service Workers by setting browser.newContext.serviceWorkers to `'block'`. It might be that you are using a mock tool such as Mock Service Worker (MSW). While this tool works out of the box for mocking responses, it adds its own Service Worker that takes over the network requests, hence making them invisible to browserContext.route() and page.route(). If you are interested in both network testing and mocking, consider using built-in browserContext.route() and page.route() for [response mocking](#handle-requests). ###### If you're interested in not solely using Service Workers for testing and network mocking, but in routing and listening for requests made by Service Workers themselves, please see [this guide](./service-workers.md).

mock#mock-api-requestsrank 2 · score 0.788not in golden set

The following code will intercept all the calls to `*/**/api/v1/fruits` and will return a custom response instead. No requests to the API will be made. The test goes to the URL that uses the mocked route and asserts that mock data is present on the page. ```js test("mocks a fruit and doesn't call api", async ({ page }) => { // Mock the api call before navigating await page.route('*/**/api/v1/fruits', async route => { const json = [{ name: 'Strawberry', id: 21 }]; await route.fulfill({ json }); }); // Go to the page await page.goto('https://demo.playwright.dev/api-mocking'); // Assert that the Strawberry fruit is visible await expect(page.getByText('Strawberry')).toBeVisible(); }); ``` You can see from the trace of the example test that the API was never called, it was however fulfilled with the mock data. <img src="https://github.com/microsoft/playwright/assets/13063165/3dc14cbf-c100-4efc-ac21-d7b52d698b53" alt="api mocking trace" width="2946" height="1902" /> Read more about [advanced networking](./network.md).

network#handle-requestsrank 3 · score 0.774not in golden set

```js await page.route('**/api/fetch_data', route => route.fulfill({ status: 200, body: testData, })); await page.goto('https://example.com'); ``` You can mock API endpoints via handling the network requests in your Playwright script.

Faithfulness

Heuristic grounding score 0.83 over 2 claim(s).

failnet-wait-for-responseHow do I wait for a specific API response before asserting on the page?0.85

Answer

You can wait for a specific network response by using `page.waitForResponse()` with a glob URL pattern, regular expression, or predicate before performing an action like clicking a button [network#variations, network#network-events]. Then, await the resulting response promise before making assertions [network#variations, network#network-events].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 2 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 2/2 expected chunk(s)
fail	grounding	0.27	2 unsupported claim(s): You can wait for a specific network response by using `page.waitForResponse()` with a glob
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.667
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

network#variationsrank 1 · score 0.812expected

Wait for Responses with page.waitForResponse() ```js // Use a RegExp. Note no await. const responsePromise = page.waitForResponse(/\.jpeg$/); await page.getByText('Update').click(); const response = await responsePromise; // Use a predicate taking a Response object. Note no await. const responsePromise = page.waitForResponse(response => response.url().includes(token)); await page.getByText('Update').click(); const response = await responsePromise; ```

network#network-eventsrank 2 · score 0.811expected

You can monitor all the Requests and Responses: ```js // Subscribe to 'request' and 'response' events. page.on('request', request => console.log('>>', request.method(), request.url())); page.on('response', response => console.log('<<', response.status(), response.url())); await page.goto('https://example.com'); ``` Or wait for a network response after the button click with page.waitForResponse(): ```js // Use a glob URL pattern. Note no await. const responsePromise = page.waitForResponse('**/api/fetch_data'); await page.getByText('Update').click(); const response = await responsePromise; ```

mock_browser#creating-mocksrank 3 · score 0.805not in golden set

Since the page may be calling the API very early while loading it's important to setup all the mocks before the page started loading. The easiest way to achieve that is to call page.addInitScript(): ```js await page.addInitScript(() => { const mockBattery = { level: 0.75, charging: true, chargingTime: 1800, dischargingTime: Infinity, addEventListener: () => { } }; // Override the method to always return mock battery info. window.navigator.getBattery = async () => mockBattery; }); ``` Once this is done you can navigate the page and check its UI state: ```js // Configure mock API before each test. test.beforeEach(async ({ page }) => { await page.addInitScript(() => { const mockBattery = { level: 0.90, charging: true, chargingTime: 1800, // seconds dischargingTime: Infinity, addEventListener: () => { } }; // Override the method to always return mock battery info. window.navigator.getBattery = async () => mockBattery; }); }); test('show battery status', async ({ page }) => { await page.goto('/'); await expect(page.locator('.battery-percentage')).toHaveText('90%'); await expect(page.locator('.battery-status')).toHaveText('Adapter'); await expect(page.locator('.battery-fully')).toHaveText('00:30'); }); ```

Faithfulness

Heuristic grounding score 0.27 over 2 claim(s).

Ungrounded claim(s):

You can wait for a specific network response by using `page.waitForResponse()` with a glob URL pattern, regular expression, or predicate before performing an action like clicking a button [network#variations, network#network-events].
Then, await the resulting response promise before making assertions [network#variations, network#network-events].

passnet-offline-webrtcDoes the offline option also cut WebRTC connections?0.78

Answer

No, offline emulation does not affect WebRTC traffic, as established `RTCPeerConnection`s will keep sending and receiving media over UDP [emulation#offline].

Checks

	Check	Score	Detail
fail	exact_match	0.00	answer differs from expected_answer
fail	fuzzy_match	0.57	similarity 0.57 vs threshold 0.60
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.90	all 1 claim(s) grounded (score 0.90)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

emulation#offlinerank 1 · score 0.817expected

Emulate the network being offline. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ use: { offline: true }, }); ``` Offline emulation only affects requests that go through the browser's regular network stack, such as page navigations, `fetch()`, `XMLHttpRequest` and WebSockets. It does not affect WebRTC traffic: established `RTCPeerConnection`s keep sending and receiving media over UDP. To test WebRTC connection loss, interrupt the connection outside the browser, for example by stopping the TURN server or using an OS-level firewall.

test_use_options#network-optionsrank 2 · score 0.671not in golden set

Available options to configure networking: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ use: { // Whether to automatically download all the attachments. acceptDownloads: false, // An object containing additional HTTP headers to be sent with every request. extraHTTPHeaders: { 'X-My-Header': 'value', }, // Credentials for HTTP authentication. httpCredentials: { username: 'user', password: 'pass', }, // Whether to ignore HTTPS errors during navigation. ignoreHTTPSErrors: true, // Whether to emulate network being offline. offline: true, // Proxy settings used for all pages in the test. proxy: { server: 'http://myproxy.com:3128', bypass: 'localhost', }, }, }); ``` | Option | Description | | :- | :- | | testOptions.acceptDownloads | Whether to automatically download all the attachments, defaults to `true`. [Learn more](./downloads.md) about working with downloads. | | testOptions.extraHTTPHeaders | An object containing additional HTTP headers to be sent with every request. All header values must be strings. | | testOptions.httpCredentials | Credentials for [HTTP authentication](./network.md#http-authentication). | | testOptions.ignoreHTTPSErrors | Whether to ignore HTTPS errors during navigation. | | testOptions.offline | Whether to emulate network being offline. | | testOptions.proxy |

mock#mock-websocketsrank 3 · score 0.664not in golden set

The following code will intercept WebSocket connections and mock entire communication over the WebSocket, instead of connecting to the server. This example responds to a `"request"` with a `"response"`. ```js await page.routeWebSocket('wss://example.com/ws', ws => { ws.onMessage(message => { if (message === 'request') ws.send('response'); }); }); ``` Alternatively, you may want to connect to the actual server, but intercept messages in-between and modify or block them. Here is an example that modifies some of the messages sent by the page to the server, and leaves the rest unmodified. ```js await page.routeWebSocket('wss://example.com/ws', ws => { const server = ws.connectToServer(); ws.onMessage(message => { if (message === 'request') server.send('request2'); else server.send(message); }); }); ``` For more details, see WebSocketRoute.

Faithfulness

Heuristic grounding score 0.90 over 1 claim(s).

passnet-download-eventHow do I get hold of a file the page downloads during a test?0.93

Answer

You can obtain a downloaded file by waiting for the `download` event using `page.waitForEvent('download')` and then saving it with `download.saveAs()` [downloads#introduction]. Alternatively, you can handle the event with `page.on('download', ...)` and use `download.path()` [downloads#variations].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 2 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.64	all 2 claim(s) grounded (score 0.64)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

downloads#introductionrank 1 · score 0.799expected

For every attachment downloaded by the page, the 'download' event event is emitted. All these attachments are downloaded into a temporary folder. You can obtain the download url, file name and payload stream using the Download object from the event. You can specify where to persist downloaded files using the browserType.launch.downloadsPath option in browserType.launch(). Downloaded files are deleted when the browser context that produced them is closed. Here is the simplest way to handle the file download: ```js // Start waiting for download before clicking. Note no await. const downloadPromise = page.waitForEvent('download'); await page.getByText('Download file').click(); const download = await downloadPromise; // Wait for the download process to complete and save the downloaded file somewhere. await download.saveAs('/path/to/save/at/' + download.suggestedFilename()); ```

downloads#variationsrank 2 · score 0.760not in golden set

If you have no idea what initiates the download, you can still handle the event: ```js page.on('download', download => download.path().then(console.log)); ``` Note that handling the event forks the control flow and makes the script harder to follow. Your scenario might end while you are downloading a file since your main control flow is not awaiting for this operation to resolve. For uploading files, see the [uploading files](./input.md#upload-files) section.

aria_snapshots#snapshots-as-separate-filesrank 3 · score 0.739not in golden set

To store your snapshots in a separate file, use the `toMatchAriaSnapshot` method with the `name` option, specifying a `.aria.yml` file extension. ```js await expect(page.getByRole('main')).toMatchAriaSnapshot({ name: 'main.aria.yml' }); ``` By default, snapshots from a test file `example.spec.ts` are placed in the `example.spec.ts-snapshots` directory. As snapshots should be the same across browsers, only one snapshot is saved even if testing with multiple browsers. Should you wish, you can customize the [snapshot path template](./api/class-testconfig#test-config-snapshot-path-template) using the following configuration: ```js export default defineConfig({ expect: { toMatchAriaSnapshot: { pathTemplate: '__snapshots__/{testFilePath}/{arg}{ext}', }, }, }); ```

Faithfulness

Heuristic grounding score 0.64 over 2 claim(s).

failnet-new-tabA link opens in a new tab. How do I get a handle on that new page?0.91

Answer

You can use `context.waitForEvent('page')` to wait for and get the new page that is created in the context [pages#handling-new-pages]. Alternatively, you can listen for new pages using `context.on('page', ...)` [pages#handling-new-pages].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
fail	grounding	0.56	1 unsupported claim(s): Alternatively, you can listen for new pages using `context.on('page', ...)` [pages#handlin
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

pages#handling-new-pagesrank 1 · score 0.791expected

The `page` event on browser contexts can be used to get new pages that are created in the context. This can be used to handle new pages opened by `target="_blank"` links. ```js // Start waiting for new page before clicking. Note no await. const pagePromise = context.waitForEvent('page'); await page.getByText('open new tab').click(); const newPage = await pagePromise; // Interact with the new page normally. await newPage.getByRole('button').click(); console.log(await newPage.title()); ``` If the action that triggers the new page is unknown, the following pattern can be used. ```js // Get all new pages (including popups) in the context context.on('page', async page => { await page.waitForLoadState(); console.log(await page.title()); }); ```

pages#pagesrank 2 · score 0.726not in golden set

Each BrowserContext can have multiple pages. A Page refers to a single tab or a popup window within a browser context. It should be used to navigate to URLs and interact with the page content. ```js // Create a page. const page = await context.newPage(); // Navigate explicitly, similar to entering a URL in the browser. await page.goto('http://example.com'); // Fill an input. await page.locator('#search').fill('query'); // Navigate implicitly by clicking a link. await page.locator('#submit').click(); // Expect a new url. console.log(page.url()); ```

pages#handling-popupsrank 3 · score 0.705not in golden set

If the page opens a pop-up (e.g. pages opened by `target="_blank"` links), you can get a reference to it by listening to the `popup` event on the page. This event is emitted in addition to the `browserContext.on('page')` event, but only for popups relevant to this page. ```js // Start waiting for popup before clicking. Note no await. const popupPromise = page.waitForEvent('popup'); await page.getByText('open the popup').click(); const popup = await popupPromise; // Interact with the new popup normally. await popup.getByRole('button').click(); console.log(await popup.title()); ``` If the action that triggers the popup is unknown, the following pattern can be used. ```js // Get all popups when they open page.on('popup', async popup => { await popup.waitForLoadState(); console.log(await popup.title()); }); ```

Faithfulness

Heuristic grounding score 0.56 over 2 claim(s).

Ungrounded claim(s):

Alternatively, you can listen for new pages using `context.on('page', ...)` [pages#handling-new-pages].

failnet-dialog-defaultWhat happens to an alert() dialog if my test never handles it?0.74

Answer

By default, dialogs like `alert()` are automatically dismissed by Playwright if there is no listener for the 'dialog' event [dialogs#alert-confirm-prompt-dialogs].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
fail	keyword_presence	0.00	missing: auto-dismissed
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.71	all 1 claim(s) grounded (score 0.71)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	0.500

Retrieved chunks

test_assertions#auto-retrying-assertionsrank 1 · score 0.734not in golden set

The following assertions will retry until the assertion passes, or the assertion timeout is reached. Note that retrying assertions are async, so you must `await` them. | Assertion | Description | | :- | :- | | [await expect(locator).toBeAttached()](./api/class-locatorassertions.md#locator-assertions-to-be-attached) | Element is attached | | [await expect(locator).toBeChecked()](./api/class-locatorassertions.md#locator-assertions-to-be-checked) | Checkbox is checked | | [await expect(locator).toBeDisabled()](./api/class-locatorassertions.md#locator-assertions-to-be-disabled) | Element is disabled | | [await expect(locator).toBeEditable()](./api/class-locatorassertions.md#locator-assertions-to-be-editable) | Element is editable | | [await expect(locator).toBeEmpty()](./api/class-locatorassertions.md#locator-assertions-to-be-empty) | Container is empty | | [await expect(locator).toBeEnabled()](./api/class-locatorassertions.md#locator-assertions-to-be-enabled) | Element is enabled | | [await expect(locator).toBeFocused()](./api/class-locatorassertions.md#locator-assertions-to-be-focused) | Element is focused | | [await expect(locator).toBeHidden()](./api/class-locatorassertions.md#locator-assertions-to-be-hidden) | Element is not visible | | [await expect(locator).toBeInViewport()](./api/class-locatorassertions.md#locator-assertions-to-be-in-viewport) | Element intersects viewport | | [await expect(locator).toBeVisible()](./api/class-locatorassertions.md#locator-assertions-to-be-visible) | Element is visible | | [await expect(locator).toContainText()](./api/class-locatorassertions.md#locator-assertions-to-contain-text) | Element contains text | | [await expect(locator).toContainClass()](./api/class-locatorassertions.md#locator-assertions-to-contain-class) | Element has specified CSS classes | | [await expect(locator).toHaveAccessibleDescription()](./api/class-locatorassertions.md#locator-assertions-to-have-accessible-description) | Element has a matching [accessible description](https://w3c.github.io/accname/#dfn-accessible-description) | | [await expect(locator).toHaveAccessibleName()](./api/class-locatorassertions.md#locator-assertions-to-have-accessible-name) | Element has a matching [accessible name](https://w3c.github.io/accname/#dfn-accessible-name) | | [await expect(locator).toHaveAttribute()](./api/class-locatorassertions.md#locator-assertions-to-have-attribute) | Element has a DOM attribute | | [await expect(locator).toHaveClass()](./api/class-locatorassertions.md#locator-assertions-to-have-class) | Element has specified CSS class property | | [await

dialogs#alert-confirm-prompt-dialogsrank 2 · score 0.727expected

By default, dialogs are auto-dismissed by Playwright, so you don't have to handle them. However, you can register a dialog handler before the action that triggers the dialog to either dialog.accept() or dialog.dismiss() it. ```js page.on('dialog', dialog => dialog.accept()); await page.getByRole('button').click(); ``` the 'dialog' event listener **must handle** the dialog. Otherwise your action will stall, be it locator.click() or something else. That's because dialogs in Web are modals and therefore block further page execution until they are handled. As a result, the following snippet will never resolve: WRONG! ```js page.on('dialog', dialog => console.log(dialog.message())); await page.getByRole('button').click(); // Will hang here ``` If there is no listener for the 'dialog' event, all dialogs are automatically dismissed.

dialogs#print-dialogsrank 3 · score 0.727not in golden set

In order to assert that a print dialog via [`window.print`](https://developer.mozilla.org/en-US/docs/Web/API/Window/print) was triggered, you can use the following snippet: ```js await page.goto('<url>'); await page.evaluate('(() => {window.waitForPrintDialog = new Promise(f => window.print = f);})()'); await page.getByText('Print it!').click(); await page.waitForFunction('window.waitForPrintDialog'); ``` This will wait for the print dialog to be opened after the button is clicked. Make sure to evaluate the script before clicking the button / after the page is loaded.

Faithfulness

Heuristic grounding score 0.71 over 1 claim(s).

failtool-trace-on-ciWhat trace setting do the docs recommend for CI?0.71

Answer

The documentation recommends only recording traces for failing tests [ci_intro#viewing-the-trace-3], specifically using the `on-first-retry` setting [trace_viewer_intro#recording-a-trace].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 2 retrieved chunk(s)
fail	expected_citation_overlap	0.00	cited 0/1 expected chunk(s)
fail	grounding	0.53	grounding score 0.53 below threshold 0.55
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	0.000
precision@3	0.000
recall@3	0.000
reciprocal_rank	0.000

Retrieved chunks

trace_viewer_intro#recording-a-tracerank 1 · score 0.782not in golden set

By default the [playwright.config](./trace-viewer.md#tracing-on-ci) file contains the configuration needed to create a `trace.zip` file for each test. Traces are setup to run `on-first-retry`, meaning they run on the first retry of a failed test. Also `retries` are set to 2 when running on CI and 0 locally. This means the traces are recorded on the first retry of a failed test but not on the first run and not on the second retry. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ retries: process.env.CI ? 2 : 0, // set to 2 when running on CI // ... use: { trace: 'on-first-retry', // record traces on first retry of each test }, }); ``` To learn more about available options to record a trace check out our detailed guide on [Trace Viewer](/trace-viewer.md). Traces are normally run in a Continuous Integration (CI) environment, because locally you can use [UI Mode](/test-ui-mode.md) for developing and debugging tests. However, if you want to run traces locally without using [UI Mode](/test-ui-mode.md), you can force tracing to be on with `--trace on`. ```bash npx playwright

trace_viewer_intro#what-s-nextrank 2 · score 0.777not in golden set

- [Run tests on CI with GitHub Actions](/ci-intro.md) - [Learn more about Trace Viewer](/trace-viewer.md)

ci_intro#viewing-the-trace-3rank 3 · score 0.772not in golden set

You can upload Traces which get created on your CI like GitHub Actions as artifacts. This requires [starting and stopping the trace](./trace-viewer-intro#recording-a-trace). We recommend only recording traces for failing tests. Once your traces have been uploaded to CI, they can then be downloaded and opened using [trace.playwright.dev](https://trace.playwright.dev), which is a statically hosted variant of the Trace Viewer. You can upload trace files using drag and drop. ###### ![playwright trace viewer](./images/getting-started/trace-viewer-failed-test.png)

Expected but not retrieved: trace_viewer#tracing-on-ci

Faithfulness

Heuristic grounding score 0.53 over 1 claim(s).

failtool-open-traceHow do I open a trace.zip file that CI produced?0.70

Answer

You can download the trace file produced on CI and open it by uploading it via drag and drop at [trace.playwright.dev](https://trace.playwright.dev) [ci_intro#viewing-the-trace-3]. Alternatively, you can open a saved trace using the Playwright CLI or directly in the browser at the same website [trace_viewer#opening-trace-viewer].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
fail	keyword_presence	0.00	missing: show-trace
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 2 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
fail	grounding	0.52	1 unsupported claim(s): You can download the trace file produced on CI and open it by uploading it via drag and dr
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

trace_viewer#opening-trace-viewerrank 1 · score 0.803expected

You can open a saved trace using either the Playwright CLI or in the browser at [trace.playwright.dev](https://trace.playwright.dev). Make sure to add the full path to where your `trace.zip` file is located. ```bash npx playwright show-trace path/to/trace.zip ``` ```bash mvn exec:java -e -D exec.mainClass=com.microsoft.playwright.CLI -D exec.args="show-trace trace.zip" ``` ```bash playwright show-trace trace.zip ``` ```bash pwsh bin/Debug/netX/playwright.ps1 show-trace trace.zip ```

ci_intro#viewing-the-trace-3rank 2 · score 0.797not in golden set

You can upload Traces which get created on your CI like GitHub Actions as artifacts. This requires [starting and stopping the trace](./trace-viewer-intro#recording-a-trace). We recommend only recording traces for failing tests. Once your traces have been uploaded to CI, they can then be downloaded and opened using [trace.playwright.dev](https://trace.playwright.dev), which is a statically hosted variant of the Trace Viewer. You can upload trace files using drag and drop. ###### ![playwright trace viewer](./images/getting-started/trace-viewer-failed-test.png)

trace_viewer#recording-a-trace-3rank 3 · score 0.795not in golden set

Traces can be recorded using the browserContext.tracing API as follows: This will record the trace and place it into the file named `trace.zip`.

Faithfulness

Heuristic grounding score 0.52 over 2 claim(s).

Ungrounded claim(s):

You can download the trace file produced on CI and open it by uploading it via drag and drop at [trace.playwright.dev](https://trace.playwright.dev) [ci_intro#viewing-the-trace-3].

passtool-ui-modeWhat is the command to open UI mode?0.96

Answer

To open UI mode, run the command `npx playwright test --ui` in your terminal [test_ui_mode#opening-ui-mode].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.81	all 1 claim(s) grounded (score 0.80)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

test_ui_mode#opening-ui-moderank 1 · score 0.874expected

To open UI mode, run the following command in your terminal: ```bash npx playwright test --ui ```

test_ui_mode#docker-github-codespacesrank 2 · score 0.763not in golden set

For Docker and GitHub Codespaces environments, you can run UI mode in the browser. In order for an endpoint to be accessible outside of the container, it needs to be bound to the `0.0.0.0` interface: ```bash npx playwright test --ui-host=0.0.0.0 ``` In the case of GitHub Codespaces, the port gets [forwarded automatically](https://docs.github.com/en/codespaces/developing-in-codespaces/forwarding-ports-in-your-codespace#about-forwarded-ports), so you can open UI mode in the browser by clicking on the link in the terminal. To have a static port, you can pass the `--ui-port` flag: ```bash npx playwright test --ui-port=8080 --ui-host=0.0.0.0 ``` Be aware that when specifying the `--ui-host=0.0.0.0` flag, UI Mode with your traces, the passwords and secrets are accessible from other machines inside your network. In the case of GitHub Codespaces, the ports are only accessible from your account by default.

test_ui_mode#running-your-testsrank 3 · score 0.739not in golden set

Once you launch UI Mode you will see a list of all your test files. You can run all your tests by clicking the triangle icon in the sidebar. You can also run a single test file, a block of tests or a single test by hovering over the name and clicking on the triangle next to it. <img src="https://github.com/microsoft/playwright/assets/13063165/6b87712f-64a5-4d73-a91d-6562b864712c" alt="running tests in ui mode" width="3960" height="2326" />

Faithfulness

Heuristic grounding score 0.81 over 1 claim(s).

passtool-codegenHow do I record a test by clicking through the site instead of writing it by hand?0.75

Answer

To record a new test, click on the **Record new** button from the Testing sidebar, which creates a file and opens a browser window [codegen#record-a-new-test]. Then, go to the URL you wish to test and click around to record your user actions, which will be generated directly as test code in VS Code [codegen#record-a-new-test].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
fail	expected_citation_overlap	0.00	cited 0/2 expected chunk(s)
pass	grounding	0.74	all 2 claim(s) grounded (score 0.74)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	0.000
precision@3	0.000
recall@3	0.000
reciprocal_rank	0.000

Retrieved chunks

codegen#record-a-new-testrank 1 · score 0.771not in golden set

To record a test click on the **Record new** button from the Testing sidebar. This will create a `test-1.spec.ts` file as well as open up a browser window. <img height="1940" width="2770" alt="record new in vs code" src="https://user-images.githubusercontent.com/13063165/220961665-615d0ab8-3f0b-439c-ad0b-0424d9aa154b.png" /> In the browser go to the URL you wish to test and start clicking around to record your user actions. <img src="https://github.com/microsoft/playwright/assets/13063165/1d4c8f37-8325-4816-a665-d0e95e63f509" alt="generating user actions" width="2788" height="1824" /> Playwright will record your actions and generate the test code directly in VS Code. You can also generate assertions by choosing one of the icons in the toolbar and then clicking on an element on the page to assert against. The following assertions can be generated: * `'assert visibility'` to assert that an element is visible * `'assert text'` to assert that an element contains specific text * `'assert value'` to assert that an element has a specific value <img src="https://github.com/microsoft/playwright/assets/13063165/d131eb35-b2ca-4bf4-a8ac-88b6e40dcf07" alt="generating assertions" width="2788" height="1400" /> Once you are done recording click the **cancel** button or close the browser window. You can then inspect your `test-1.spec.ts` file and manually improve it if needed. <img

codegen#record-at-cursorrank 2 · score 0.760not in golden set

To record from a specific point in your test move your cursor to where you want to record more actions and then click the **Record at cursor** button from the Testing sidebar. If your browser window is not already open then first run the test with 'Show browser' checked and then click the **Record at cursor** button. <img src="https://github.com/microsoft/playwright/assets/13063165/77948ab8-92a2-435f-9833-0944da5ae664" alt="record at cursor in vs code" width="3126" height="1756" /> In the browser window start performing the actions you want to record. <img height="1824" width="2788" alt="add feed the dog to todo app" src="https://user-images.githubusercontent.com/13063165/220960770-6435cec7-1723-42a8-8c1f-8244e2d800c7.png" /> In the test file in VS Code you will see your new generated actions added to your test at the cursor position. <img src="https://github.com/microsoft/playwright/assets/13063165/4f4bb34e-9cda-41fe-bf65-8d8016d84c7f" alt="code from a generated test" width="3180" height="1944" />

trace_viewer#run-trace-only-on-failurerank 3 · score 0.750not in golden set

Setup your tests to record a trace only when the test fails: <Tabs groupId="test-runners" defaultValue="mstest" values={[ {label: 'MSTest', value: 'mstest'}, {label: 'NUnit', value: 'nunit'}, {label: 'xUnit', value: 'xunit'}, {label: 'xUnit v3', value: 'xunit-v3'}, ] }> <TabItem value="nunit"> </TabItem> <TabItem value="mstest"> </TabItem> <TabItem value="xunit"> </TabItem> <TabItem value="xunit-v3"> </TabItem> </Tabs>

Expected but not retrieved: codegen#running-codegen, codegen_intro#running-codegen

Faithfulness

Heuristic grounding score 0.74 over 2 claim(s).

failtool-install-one-browserHow do I install only WebKit instead of all browsers?0.00

Answer

I do not have enough information in the provided documents to answer that.

Checks

	Check	Detail
skip	exact_match	no expected_answer in dataset
skip	fuzzy_match	no expected_answer in dataset
fail	keyword_presence	missing: install webkit
skip	forbidden_absent	no forbidden_keywords in dataset
fail	refusal_behaviour	refused a question it should have answered
skip	citation_present	refusals are not expected to cite sources
fail	expected_citation_overlap	cited 0/1 expected chunk(s)
skip	grounding	skipped: refusals are not grounded in retrieved text by design
skip	llm_judge_faithfulness	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	0.000
precision@3	0.000
recall@3	0.000
reciprocal_rank	0.000

Retrieved chunks

test_cli#install-optionsrank 1 · score 0.737not in golden set

docker#alpinerank 2 · score 0.719not in golden set

Browser builds for Firefox and WebKit are built for the [glibc](https://en.wikipedia.org/wiki/Glibc) library. Alpine Linux and other distributions that are based on the [musl](https://en.wikipedia.org/wiki/Musl) standard library are not supported.

test_cli#options-3rank 3 · score 0.714not in golden set

Expected but not retrieved: browsers#install-browsers

Faithfulness

Heuristic grounding score 0.09 over 1 claim(s).

Ungrounded claim(s):

I do not have enough information in the provided documents to answer that.

passtool-docker-imageDoes the official Docker image include the Playwright npm package itself?0.75

Answer

No, the Playwright package/dependency is not included in the Docker image and should be installed separately [docker#introduction].

Checks

	Check	Score	Detail
fail	exact_match	0.00	answer differs from expected_answer
fail	fuzzy_match	0.44	similarity 0.44 vs threshold 0.60
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
pass	grounding	0.84	all 1 claim(s) grounded (score 0.84)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	0.500

Retrieved chunks

docker#image-tagsrank 1 · score 0.856not in golden set

See [all available image tags]. We currently publish images with the following tags: - `:v%%VERSION%%` - Playwright v%%VERSION%% release docker image based on Ubuntu 24.04 LTS (Noble Numbat). - `:v%%VERSION%%-noble` - Playwright v%%VERSION%% release docker image based on Ubuntu 24.04 LTS (Noble Numbat). - `:v%%VERSION%%-jammy` - Playwright v%%VERSION%% release docker image based on Ubuntu 22.04 LTS (Jammy Jellyfish). - `:v%%VERSION%%-resolute` - Playwright v%%VERSION%% release docker image based on Ubuntu 26.04 LTS (Resolute Raccoon). It is recommended to always pin your Docker image to a specific version if possible. If the Playwright version in your Docker image does not match the version in your project/tests, Playwright will be unable to locate browser executables.

docker#introductionrank 2 · score 0.843expected

[Dockerfile.noble] can be used to run Playwright scripts in Docker environment. This image includes the [Playwright browsers](./browsers.md#install-browsers) and [browser system dependencies](./browsers.md#install-system-dependencies). The Playwright package/dependency is not included in the image and should be installed separately.

docker#build-your-own-imagerank 3 · score 0.831not in golden set

To run Playwright inside Docker, you need to have Node.js, [Playwright browsers](./browsers.md#install-browsers) and [browser system dependencies](./browsers.md#install-system-dependencies) installed. See the following Dockerfile:

Faithfulness

Heuristic grounding score 0.84 over 1 claim(s).

passauth-shared-accountWhat is the recommended way to log in once and reuse the session across all tests?0.88

Answer

The recommended approach is to authenticate once in a setup project, save the authentication state, and then reuse it to bootstrap each test already authenticated [auth#basic-shared-account-in-all-tests]. You can also reuse the signed-in state in tests by using the setup project to log in only once and skip the log-in step for all tests [best_practices#make-tests-as-isolated-as-possible~2].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 2 retrieved chunk(s)
pass	expected_citation_overlap	0.50	cited 1/2 expected chunk(s)
pass	grounding	0.89	all 2 claim(s) grounded (score 0.89)
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	0.500
reciprocal_rank	0.333

Retrieved chunks

best_practices#make-tests-as-isolated-as-possible~2rank 1 · score 0.793not in golden set

await page.goto('https://github.com/login'); await page.getByLabel('Username or email address').fill('username'); await page.getByLabel('Password').fill('password'); await page.getByRole('button', { name: 'Sign in' }).click(); }); test('first', async ({ page }) => { // page is signed in. }); test('second', async ({ page }) => { // page is signed in. }); ``` You can also reuse the signed-in state in the tests with [setup project](./auth.md#basic-shared-account-in-all-tests). That way you can log in only once and then skip the log in step for all of the tests.

best_practices#make-tests-as-isolated-as-possiblerank 2 · score 0.780not in golden set

Each test should be completely isolated from another test and should run independently with its own local storage, session storage, data, cookies etc. [Test isolation](./browser-contexts.md) improves reproducibility, makes debugging easier and prevents cascading test failures. In order to avoid repetition for a particular part of your test you can use [before and after hooks](/api/class-test.md). Within your test file add a before hook to run a part of your test before each test such as going to a particular URL or logging in to a part of your app. This keeps your tests isolated as no test relies on another. However it is also ok to have a little duplication when tests are simple enough especially if it keeps your tests clearer and easier to read and maintain. ```js import { test } from '@playwright/test'; test.beforeEach(async ({ page }) => { // Runs before each test and signs in each page. await page.goto('https://github.com/login'); await page.getByLabel('Username or email address').fill('username'); await page.getByLabel('Password').fill('password'); await page.getByRole('button', { name: 'Sign in' }).click(); }); test('first', async ({ page }) => { // page is signed in. });

auth#basic-shared-account-in-all-testsrank 3 · score 0.778expected

This is the **recommended** approach for tests **without server-side state**. Authenticate once in the **setup project**, save the authentication state, and then reuse it to bootstrap each test already authenticated. **When to use** - When you can imagine all your tests running at the same time with the same account, without affecting each other. **When not to use** - Your tests modify server-side state. For example, one test checks the rendering of the settings page, while the other test is changing the setting, and you run tests in parallel. In this case, tests must use different accounts. - Your authentication is browser-specific. **Details** Create `tests/auth.setup.ts` that will prepare authenticated browser state for all other tests. ```js import { test as setup, expect } from '@playwright/test'; import path from 'path'; const authFile = path.join(__dirname, '../playwright/.auth/user.json'); setup('authenticate', async ({ page }) => { // Perform authentication steps. Replace these actions with your own. await page.goto('https://github.com/login'); await page.getByLabel('Username or email address').fill('username'); await page.getByLabel('Password').fill('password'); await page.getByRole('button', { name: 'Sign in' }).click(); // Wait until the page receives the cookies. // // Sometimes login flow

Expected but not retrieved: auth#core-concepts

Faithfulness

Heuristic grounding score 0.89 over 2 claim(s).

failauth-per-workerMy tests change server-side data, so a single shared login breaks them in parallel. What does the guide suggest?0.93

Answer

The guide suggests using the "Moderate: one account per parallel worker" approach, where each parallel worker is authenticated once using a unique account [auth#moderate-one-account-per-parallel-worker]. This is recommended when tests modify shared server-side state [auth#moderate-one-account-per-parallel-worker].

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
pass	keyword_presence	1.00	all 1 required keyword(s) present
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly answered
pass	citation_present	1.00	cites 1 retrieved chunk(s)
pass	expected_citation_overlap	1.00	cited 1/1 expected chunk(s)
fail	grounding	0.64	1 unsupported claim(s): The guide suggests using the "Moderate: one account per parallel worker" approach, where e
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	1.000
precision@3	0.333
recall@3	1.000
reciprocal_rank	1.000

Retrieved chunks

auth#moderate-one-account-per-parallel-workerrank 1 · score 0.780expected

This is the **recommended** approach for tests that **modify server-side state**. In Playwright, worker processes run in parallel. In this approach, each parallel worker is authenticated once. All tests ran by worker are reusing the same authentication state. We will need multiple testing accounts, one per each parallel worker. **When to use** - Your tests modify shared server-side state. For example, one test checks the rendering of the settings page, while the other test is changing the setting. **When not to use** - Your tests do not modify any shared server-side state. In this case, all tests can use a single shared account. **Details** We will authenticate once per [worker process](./test-parallel.md#worker-processes), each with a unique account. Create `playwright/fixtures.ts` file that will [override `storageState` fixture](./test-fixtures.md#overriding-fixtures) to authenticate once per worker. Use testInfo.parallelIndex to differentiate between workers. ```js import { test as baseTest, expect } from '@playwright/test'; import fs from 'fs'; import path from 'path'; export * from '@playwright/test'; export const test = baseTest.extend<{}, { workerStorageState: string }>({ // Use the same storage state for all tests in this worker. storageState: ({

auth#authenticate-with-api-request~2rank 2 · score 0.777not in golden set

workerStorageState: [async ({}, use) => { // Use parallelIndex as a unique identifier for each worker. const id = test.info().parallelIndex; const fileName = path.resolve(test.info().project.outputDir, `.auth/${id}.json`); if (fs.existsSync(fileName)) { // Reuse existing authentication state if any. await use(fileName); return; } // Important: make sure we authenticate in a clean environment by unsetting storage state. const context = await request.newContext({ storageState: undefined }); // Acquire a unique account, for example create a new one. // Alternatively, you can have a list of precreated accounts for testing. // Make sure that accounts are unique, so that multiple team members // can run tests at the same time without interference. const account = await acquireAccount(id); // Send authentication request. Replace with your own. await context.post('https://github.com/login', { form: { 'user': 'user', 'password': 'password' } }); await context.storageState({ path: fileName }); await context.dispose(); await use(fileName); }, { scope: 'worker' }], }); ```

auth#basic-shared-account-in-all-testsrank 3 · score 0.775not in golden set

This is the **recommended** approach for tests **without server-side state**. Authenticate once in the **setup project**, save the authentication state, and then reuse it to bootstrap each test already authenticated. **When to use** - When you can imagine all your tests running at the same time with the same account, without affecting each other. **When not to use** - Your tests modify server-side state. For example, one test checks the rendering of the settings page, while the other test is changing the setting, and you run tests in parallel. In this case, tests must use different accounts. - Your authentication is browser-specific. **Details** Create `tests/auth.setup.ts` that will prepare authenticated browser state for all other tests. ```js import { test as setup, expect } from '@playwright/test'; import path from 'path'; const authFile = path.join(__dirname, '../playwright/.auth/user.json'); setup('authenticate', async ({ page }) => { // Perform authentication steps. Replace these actions with your own. await page.goto('https://github.com/login'); await page.getByLabel('Username or email address').fill('username'); await page.getByLabel('Password').fill('password'); await page.getByRole('button', { name: 'Sign in' }).click(); // Wait until the page receives the cookies. // // Sometimes login flow

Faithfulness

Heuristic grounding score 0.64 over 2 claim(s).

Ungrounded claim(s):

The guide suggests using the "Moderate: one account per parallel worker" approach, where each parallel worker is authenticated once using a unique account [auth#moderate-one-account-per-parallel-worker].

passoos-pricingHow much does the Playwright enterprise licence cost per seat?1.00

Answer

I do not have enough information in the provided documents to answer that.

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
skip	keyword_presence	0.00	no required_keywords in dataset
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly refused
skip	citation_present	0.00	refusals are not expected to cite sources
skip	expected_citation_overlap	0.00	no expected_chunks in dataset
skip	grounding	0.00	skipped: refusals are not grounded in retrieved text by design
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	n/a
precision@3	n/a
recall@3	n/a
reciprocal_rank	n/a

Retrieved chunks

clock#related-videosrank 1 · score 0.719not in golden set

intro#installing-playwrightrank 2 · score 0.674not in golden set

Get started by installing Playwright using one of the following methods.

debug#playwright-selectorrank 3 · score 0.667not in golden set

Query the Playwright selector, using the actual Playwright query engine, for example: ```bash playwright.$('.auth-form >> text=Log in'); <button>Log in</button> ```

Faithfulness

Heuristic grounding score 0.00 over 1 claim(s).

Ungrounded claim(s):

I do not have enough information in the provided documents to answer that.

passoos-cypress-retriesHow do I configure retries in Cypress?1.00

Answer

I do not have enough information in the provided documents to answer that.

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
skip	keyword_presence	0.00	no required_keywords in dataset
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly refused
skip	citation_present	0.00	refusals are not expected to cite sources
skip	expected_citation_overlap	0.00	no expected_chunks in dataset
skip	grounding	0.00	skipped: refusals are not grounded in retrieved text by design
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	n/a
precision@3	n/a
recall@3	n/a
reciprocal_rank	n/a

Retrieved chunks

test_retries#retries~2rank 1 · score 0.714not in golden set

passed (4s) ``` You can detect retries at runtime with testInfo.retry, which is accessible to any test, hook or fixture. Here is an example that clears some server-side state before a retry. ```js import { test, expect } from '@playwright/test'; test('my test', async ({ page }, testInfo) => { if (testInfo.retry) await cleanSomeCachesOnTheServer(); // ... }); ``` You can specify retries for a specific group of tests or a single file with test.describe.configure(). ```js import { test, expect } from '@playwright/test'; test.describe(() => { // All tests in this describe group will get 2 retry attempts. test.describe.configure({ retries: 2 }); test('test 1', async ({ page }) => { // ... }); test('test 2', async ({ page }) => { // ... }); }); ```

test_configuration#basic-configurationrank 2 · score 0.704not in golden set

Here are some of the most common configuration options. ```js import { defineConfig, devices } from '@playwright/test'; export default defineConfig({ // Look for test files in the "tests" directory, relative to this configuration file. testDir: 'tests', // Run all tests in parallel. fullyParallel: true, // Fail the build on CI if you accidentally left test.only in the source code. forbidOnly: !!process.env.CI, // Retry on CI only. retries: process.env.CI ? 2 : 0, // Opt out of parallel tests on CI. workers: process.env.CI ? 1 : undefined, // Limit the whole test run, so that it fails with a report instead of hanging. globalTimeout: 60 * 60 * 1000, // Reporter to use reporter: 'html', use: { // Base URL to use in actions like `await page.goto('/')`. baseURL: 'http://localhost:3000', // Collect trace when retrying the failed test. trace: 'on-first-retry', }, // Configure projects for major browsers. projects: [ { name: 'chromium', use: { ...devices['Desktop Chrome'] }, }, ], // Run your local dev server before starting the tests. webServer: { command: 'npm run start', url: 'http://localhost:3000', reuseExistingServer: !process.env.CI, }, }); ```

test_global_setup_teardown#examplerank 3 · score 0.686not in golden set

Here is a global setup example that authenticates once and reuses authentication state in tests. It uses the `baseURL` and `storageState` options from the configuration file. ```js import { chromium, type FullConfig } from '@playwright/test'; async function globalSetup(config: FullConfig) { const { baseURL, storageState } = config.projects[0].use; const browser = await chromium.launch(); const page = await browser.newPage(); await page.goto(baseURL!); await page.getByLabel('User Name').fill('user'); await page.getByLabel('Password').fill('password'); await page.getByText('Sign in').click(); await page.context().storageState({ path: storageState as string }); await browser.close(); } export default globalSetup; ``` Specify `globalSetup`, `baseURL` and `storageState` in the configuration file. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ globalSetup: require.resolve('./global-setup'), use: { baseURL: 'http://localhost:3000/', storageState: 'state.json', }, }); ``` Tests start already authenticated because we specify `storageState` that was populated by global setup. ```js import { test } from '@playwright/test'; test('test', async ({ page }) => { await page.goto('/'); // You are signed in! }); ``` You can make arbitrary data available in your tests from your global setup file by setting them as environment variables via `process.env`. ```js import type { FullConfig } from '@playwright/test'; async function

Faithfulness

Heuristic grounding score 0.00 over 1 claim(s).

Ungrounded claim(s):

I do not have enough information in the provided documents to answer that.

passoos-selenium-grid-javaWrite me a Java Selenium test that logs in to my app.1.00

Answer

I do not have enough information in the provided documents to answer that.

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
skip	keyword_presence	0.00	no required_keywords in dataset
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly refused
skip	citation_present	0.00	refusals are not expected to cite sources
skip	expected_citation_overlap	0.00	no expected_chunks in dataset
skip	grounding	0.00	skipped: refusals are not grounded in retrieved text by design
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	n/a
precision@3	n/a
recall@3	n/a
reciprocal_rank	n/a

Retrieved chunks

auth#basic-shared-account-in-all-tests~3rank 1 · score 0.736not in golden set

testMatch: /.*\.setup\.ts/ }, { name: 'chromium', use: { ...devices['Desktop Chrome'], // Use prepared auth state. storageState: 'playwright/.auth/user.json', }, dependencies: ['setup'], }, { name: 'firefox', use: { ...devices['Desktop Firefox'], // Use prepared auth state. storageState: 'playwright/.auth/user.json', }, dependencies: ['setup'], }, ], }); ``` Tests start already authenticated because we specified `storageState` in the config. ```js import { test } from '@playwright/test'; test('test', async ({ page }) => { // page is authenticated }); ``` Note that you need to delete the stored state when it expires. If you don't need to keep the state between test runs, write the browser state under testProject.outputDir, which is automatically cleaned up before every test run.

codegen_intro#generating-locatorsrank 2 · score 0.718not in golden set

You can generate [locators](/locators.md) with the test generator. * Press the `'Record'` button to stop recording and the `'Pick Locator'` button will appear * Click the `'Pick Locator'` button and hover over elements in the browser window to see the locator highlighted underneath each element * Click the element you want to locate and the code for that locator will appear in the locator playground next to the Pick Locator button * Edit the locator in the locator playground to fine-tune it and see the matching element highlighted in the browser window * Use the copy button to copy the locator and paste it into your code ###### ![picking a locator](./images/getting-started/pick-locator-js.png) ###### ![picking a locator](./images/getting-started/pick-locator-java.png) ###### ![picking a locator](./images/getting-started/pick-locator-python.png) ###### ![picking a locator](./images/getting-started/pick-locator-csharp.png)

intro#what-s-nextrank 3 · score 0.716not in golden set

- [Write tests using web-first assertions, fixtures and locators](./writing-tests.md) - [Run single or multiple tests; headed mode](./running-tests.md) - [Generate tests with Codegen](./codegen-intro.md) - [View a trace of your tests](./trace-viewer-intro.md)

Faithfulness

Heuristic grounding score 0.00 over 1 claim(s).

Ungrounded claim(s):

I do not have enough information in the provided documents to answer that.

passoos-python-syntaxWhat is the Python syntax for page.get_by_role?1.00

Answer

I do not have enough information in the provided documents to answer that.

Checks

	Check	Score	Detail
skip	exact_match	0.00	no expected_answer in dataset
skip	fuzzy_match	0.00	no expected_answer in dataset
skip	keyword_presence	0.00	no required_keywords in dataset
skip	forbidden_absent	0.00	no forbidden_keywords in dataset
pass	refusal_behaviour	1.00	correctly refused
skip	citation_present	0.00	refusals are not expected to cite sources
skip	expected_citation_overlap	0.00	no expected_chunks in dataset
skip	grounding	0.00	skipped: refusals are not grounded in retrieved text by design
skip	llm_judge_faithfulness	0.00	skipped: judge disabled

Retrieval metrics

Metric	Value
hit_rate@3	n/a
precision@3	n/a
recall@3	n/a
reciprocal_rank	n/a

Retrieved chunks

locators#locate-by-rolerank 1 · score 0.747not in golden set

The page.getByRole() locator reflects how users and assistive technology perceive the page, for example whether some element is a button or a checkbox. When locating by role, you should usually pass the accessible name as well, so that the locator pinpoints the exact element. For example, consider the following DOM structure. ```html <h3>Sign up</h3> <label> <input type="checkbox" /> Subscribe </label> <br/> <button>Submit</button> ``` You can locate each element by its implicit role: ```js await expect(page.getByRole('heading', { name: 'Sign up' })).toBeVisible(); await page.getByRole('checkbox', { name: 'Subscribe' }).check(); await page.getByRole('button', { name: /submit/i }).click(); ``` Role locators include [buttons, checkboxes, headings, links, lists, tables, and many more](https://www.w3.org/TR/html-aria/#docconformance) and follow W3C specifications for [ARIA role](https://www.w3.org/TR/wai-aria-1.2/#roles), [ARIA attributes](https://www.w3.org/TR/wai-aria-1.2/#aria-attributes) and [accessible name](https://w3c.github.io/accname/#dfn-accessible-name). Note that many html elements like `<button>` have an [implicitly defined role](https://w3c.github.io/html-aam/#html-element-role-mappings) that is recognized by the role locator. Note that role locators **do not replace** accessibility audits and conformance tests, but rather give early feedback about the ARIA guidelines. We recommend prioritizing role locators to locate elements, as it is the closest way to how users and assistive technology perceive the page.

auth#testing-multiple-roles-with-pom-fixturesrank 2 · score 0.725not in golden set

**When to use** - You need to test how multiple authenticated roles interact together, in a single test. **Details** You can introduce fixtures that will provide a page authenticated as each role. Below is an example that [creates fixtures](./test-fixtures.md#creating-a-fixture) for two [Page Object Models](./pom.md) - admin POM and user POM. It assumes `adminStorageState.json` and `userStorageState.json` files were created in the global setup. ```js import { test as base, type Page, type Locator } from '@playwright/test'; // Page Object Model for the "admin" page. // Here you can add locators and helper methods specific to the admin page. class AdminPage { // Page signed in as "admin". page: Page; // Example locator pointing to "Welcome, Admin" greeting. greeting: Locator; constructor(page: Page) { this.page = page; this.greeting = page.locator('#greeting'); } } // Page Object Model for the "user" page. // Here you can add locators and helper methods specific to the user page. class UserPage { // Page signed in as "user". page: Page; // Example locator pointing to "Welcome, User" greeting. greeting: Locator; constructor(page: Page) { this.page = page; this.greeting =

locators#matching-two-locators-simultaneouslyrank 3 · score 0.718not in golden set

Method locator.and() narrows down an existing locator by matching an additional locator. For example, you can combine page.getByRole() and page.getByTitle() to match by both role and title. ```js const button = page.getByRole('button').and(page.getByTitle('Subscribe')); ```

Faithfulness

Heuristic grounding score 0.22 over 1 claim(s).

Ungrounded claim(s):

I do not have enough information in the provided documents to answer that.

regressedauth-per-workerMy tests change server-side data, so a single shared login breaks them in parallel. What does the guide suggest?-0.073

case went from passing to failing

Now failing: grounding

Baseline answer

We will need multiple testing accounts, one per each parallel worker. **When to use** - Your tests modify shared server-side state. [auth#moderate-one-account-per-parallel-worker] For example, one test checks the rendering of the settings page, while the other test is changing the setting. **When not to use** - Your tests do not modify any shared server-side state.

Current answer

The guide suggests using the "Moderate: one account per parallel worker" approach, where each parallel worker is authenticated once using a unique account [auth#moderate-one-account-per-parallel-worker]. This is recommended when tests modify shared server-side state [auth#moderate-one-account-per-parallel-worker].

regressedcfg-default-reporter-ciWhich reporter is used by default on CI?-0.148

case went from passing to failing

Now failing: grounding

Baseline answer

This is the default on CI. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Concise 'dot' for CI, default 'list' when running locally reporter: process.env.CI ? 'dot' : 'list', }); ``` [test_reporters#reporters-on-ci] Can be used to specify additional configuration for the output report | | `--reporter <reporter>` | Reporter to use, comma-separated, can be "list", "line", "dot", "json", "junit", "null", "github", "html", "blob" (default: "list" locally and "dot" on CI) | [test_cli#options-4]

Current answer

The `dot` reporter is used by default on CI [test_cli#options-4, test_reporters#reporters-on-ci].

regressedcfg-html-report-openHow do I stop the HTML report from opening in a browser automatically after a failed run?+0.066

case went from passing to failing

Now failing: grounding

Now passing: expected_citation_overlap

Baseline answer

Playwright Trace Viewer is a GUI tool that lets you explore recorded Playwright traces of your tests, meaning you can go back and forward through each action of your test and visually see what was happening during each action. **You will learn** - [How to record a trace](/trace-viewer-intro.md#recording-a-trace) - [How to open the HTML report](/trace-viewer-intro.md#opening-the-html-report) - [How to open and view the trace](/trace-viewer-intro.md#opening-the-trace) <LiteYouTube id="yP6AnTxC34s" title="Viewing Playwright Traces" /> [trace_viewer_intro#introduction] By default, the HTML report opens automatically if some tests failed, otherwise you can open it with the following command. ```bash npx playwright show-report ``` ![HTML Report](./images/getting-started/html-report.png) You can filter and search for tests as well as click on each test to see the test errors and explore each step of the test. ![HTML Reporter detail view](./images/getting-started/html-report-detail.png) [running_tests#test-reports]

Current answer

You can stop the HTML report from opening automatically by setting the `open` property to `'never'` in the Playwright config [test_reporters#html-reporter].

regressedcfg-max-diff-pixelsHow do I allow a small number of differing pixels in toHaveScreenshot?+0.021

case went from passing to failing

Now failing: grounding

Now passing: expected_citation_overlap

Baseline answer

Configuration for the expect assertion library. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ expect: { // Maximum time expect() should wait for the condition to be met. timeout: 5000, toHaveScreenshot: { // An acceptable amount of pixels that could be different, unset by default. maxDiffPixels: 10, }, toMatchSnapshot: { // An acceptable ratio of pixels that are different to the // total amount of pixels, between 0 and 1. maxDiffPixelRatio: 0.1, }, }, }); ``` | Option | Description | | :- | :- | | testConfig.expect | [Web first assertions](./test-assertions.md) like `expect(locator).toHaveText()` have a separate timeout of 5 seconds by default. [test_configuration#expect-options] Learn more about [test and expect timeouts](./test-timeouts.md) and how to set them for a single test. | | pageAssertions.toHaveScreenshot#1() | Configuration for the `expect(locator).toHaveScreenshot()` method. | | snapshotAssertions.toMatchSnapshot#1()| Configuration for the `expect(locator).toMatchSnapshot()` method.|

Current answer

You can allow a small number of differing pixels by using the `maxDiffPixels` option in `toHaveScreenshot` or by specifying it globally in your Playwright config [test_snapshots#maxdiffpixels].

regressedcfg-project-dependenciesHow do I make a login setup step run before my chromium and firefox projects?-0.492

case went from passing to failing

Now failing: expected_citation_overlap, grounding, keyword_presence

Baseline answer

In this example the chromium, firefox and webkit projects depend on the setup project. ```js import { defineConfig, devices } from '@playwright/test'; export default defineConfig({ projects: [ { name: 'setup', testMatch: '**/*.setup.ts', }, { name: 'chromium', use: { ...devices['Desktop Chrome'] }, dependencies: ['setup'], }, { name: 'firefox', use: { ...devices['Desktop Firefox'] }, dependencies: ['setup'], }, { name: 'webkit', use: { ...devices['Desktop Safari'] }, dependencies: ['setup'], }, ], }); ``` [test_projects#dependencies] All testing projects should use the authenticated state as `storageState`. ```js import { defineConfig, devices } from '@playwright/test'; export default defineConfig({ projects: [ // Setup project { name: 'setup', testMatch: /.*\.setup\.ts/ }, { name: 'chromium', use: { ...devices['Desktop Chrome'], // Use prepared auth state. storageState: 'playwright/.auth/user.json', }, dependencies: ['setup'], }, { name: 'firefox', use: { ...devices['Desktop Firefox'], // Use [auth#basic-shared-account-in-all-tests~2]

Current answer

You can create a new `setup` project in your config and declare it as a dependency for your chromium and firefox testing projects [auth#basic-shared-account-in-all-tests~2]. Additionally, configure the setup project with a `testMatch` pattern (such as `/.*\.setup\.ts/`) so that it always runs before all the tests [auth#basic-shared-account-in-all-tests~2, auth#basic-shared-account-in-all-tests~3].

regressedcfg-video-retainHow do I record video only for tests that fail?-0.091

case went from passing to failing

Now failing: grounding

Baseline answer

Playwright Test can record videos for your tests, controlled by the `video` option in your Playwright config. [videos#record-video] By default videos are off. - `'off'` - Do not record video. - `'on'` - Record video for each test. - `'retain-on-failure'` - Record video for each test, but remove all videos from successful test runs. - `'on-first-retry'` - Record video only when retrying a test for the first time.

Current answer

To record video only for tests that fail, you can use the `'retain-on-failure'` option for the `video` setting in your Playwright configuration [videos#record-video].

regressedcfg-worker-fixtureHow do I set something up once per worker instead of once per test?-0.994

case went from passing to failing

Now failing: expected_citation_overlap, keyword_presence, refusal_behaviour

Baseline answer

Similar to how test fixtures are set up for individual test runs, worker fixtures are set up for each worker process. [test_fixtures#worker-scoped-fixtures] Note the tuple-like syntax for the worker fixture - we have to pass `{scope: 'worker'}` so that test runner sets this fixture up once per worker.

Current answer

I do not have enough information in the provided documents to answer that.

regressedloc-manual-assertionIs expect(await locator.isVisible()).toBe(true) a good assertion?-0.109

case went from passing to failing

Now failing: grounding

Baseline answer

When using assertions such as `isVisible()` the test won't wait a single second, it will just check the locator is there and return immediately. ```js // 👎 expect(await page.getByText('welcome').isVisible()).toBe(true); ``` Use web first assertions such as `toBeVisible()` instead. ```js // 👍 await expect(page.getByText('welcome')).toBeVisible(); ``` [best_practices#don-t-use-manual-assertions] If the alert message takes half a second to appear, assertions such as `toBeVisible()` will wait and retry if needed. ```js // 👍 await expect(page.getByText('welcome')).toBeVisible(); // 👎 expect(await page.getByText('welcome').isVisible()).toBe(true); ``` [best_practices#use-web-first-assertions]

Current answer

No, that is a manual assertion and is not recommended because the test will not wait for the element and will return immediately [best_practices#don-t-use-manual-assertions]. Instead, you should use web-first assertions like `await expect(locator).toBeVisible()` [best_practices#don-t-use-manual-assertions].

regressedloc-recommended-locatorWhich locator does Playwright recommend I use first when writing resilient tests?-0.333

case went from passing to failing

Now failing: grounding, keyword_presence

Baseline answer

This guide should help you to make sure you are following our best practices and writing tests that are more resilient. [best_practices#introduction] ```js await page.getByRole('button').count(); ``` You can explicitly opt-out from strictness check by telling Playwright which element to use when multiple elements match, through locator.first(), locator.last(), and locator.nth(). [locators#works-fine-with-multiple-elements]

Current answer

Playwright prioritizes role, text, and test id locators when figuring out the best locator [codegen#introduction, best_practices#generate-locators, debug#picking-locators].

regressedloc-stable-definitionWhat does Playwright mean when it waits for an element to be "stable"?-0.478

case went from passing to failing

Now failing: expected_citation_overlap, keyword_presence

Baseline answer

Element is considered stable when it has maintained the same bounding box for at least two consecutive animation frames. [actionability#stable] For example, for locator.click(), Playwright will ensure that: - locator resolves to exactly one element - element is Visible - element is Stable, as in not animating or completed animation - element [Receives Events], as in not obscured by other elements - element is Enabled Here is the complete list of actionability checks performed for each action: | Action | Visible | Stable | [Receives Events] | Enabled | Editable | | :- | :-: | :-: | :-: | :-: | :-: | | locator.check() | Yes | Yes | Yes | Yes | - | | locator.click() | Yes | Yes | Yes | Yes | - | | locator.dblclick() | Yes | Yes | Yes | Yes | - | | locator.setChecked() | Yes | [actionability#introduction]

Current answer

Playwright considers an element to be stable when it is not animating or has completed its animation [actionability#introduction].

regressedloc-visible-opacityDoes Playwright treat an element with opacity 0 as visible?-0.732

case went from passing to failing

Now failing: expected_citation_overlap, keyword_presence, refusal_behaviour

Baseline answer

Element is considered visible when it has non-empty bounding box and does not have `visibility:hidden` computed style. [actionability#visible] Note that according to this definition: * Elements of zero size **are not** considered visible. * Elements with `display:none` **are not** considered visible. * Elements with `opacity:0` **are** considered visible.

Current answer

I do not have enough information in the provided documents to answer that.

regressednet-mock-apiHow do I return fake JSON for an API call so the test never hits the real backend?-0.121

case went from passing to failing

Now failing: grounding

Baseline answer

A common question when adding accessibility tests to an application is "how do I suppress known violations?" The following examples demonstrate a few techniques you can use. [accessibility_testing#handling-known-issues] The test goes to the URL that uses the mocked route and asserts that mock data is present on the page. ```js test("mocks a fruit and doesn't call api", async ({ page }) => { // Mock the api call before navigating await page.route('*/**/api/v1/fruits', async route => { const json = [{ name: 'Strawberry', id: 21 }]; await route.fulfill({ json }); }); // Go to the page await page.goto('https://demo.playwright.dev/api-mocking'); // Assert that the Strawberry fruit is visible await expect(page.getByText('Strawberry')).toBeVisible(); }); ``` You can see from the trace of the example test that the API was never called, it was however fulfilled with the mock data. <img src="https://github.com/microsoft/playwright/assets/13063165/3dc14cbf-c100-4efc-ac21-d7b52d698b53" alt="api mocking trace" width="2946" height="1902" /> Read more about [advanced networking](./network.md). [mock#mock-api-requests]

Current answer

To return fake JSON for an API call without hitting the real backend, you can use `page.route` to intercept the call and fulfill it with custom JSON data [mock#mock-api-requests]. For example, call `await route.fulfill({ json })` within the route handler before navigating to the page [mock#mock-api-requests].

regressednet-new-tabA link opens in a new tab. How do I get a handle on that new page?-0.088

case went from passing to failing

Now failing: grounding

Baseline answer

Note no await. const pagePromise = context.waitForEvent('page'); await page.getByText('open new tab').click(); const newPage = await pagePromise; // Interact with the new page normally. await newPage.getByRole('button').click(); console.log(await newPage.title()); ``` If the action that triggers the new page is unknown, the following pattern can be used. ```js // Get all new pages (including popups) in the context context.on('page', async page => { await page.waitForLoadState(); console.log(await page.title()); }); ``` [pages#handling-new-pages] Take a look at the following example to see how to write a test. ```js import { test, expect } from '@playwright/test'; test('has title', async ({ page }) => { await page.goto('https://playwright.dev/'); // Expect a title "to contain" a substring. await expect(page).toHaveTitle(/Playwright/); }); test('get started link', async ({ page }) => { await page.goto('https://playwright.dev/'); // Click the get started link. await page.getByRole('link', { name: 'Get started' }).click(); // Expects page to have a heading with the name of Installation. await expect(page.getByRole('heading', { name: 'Installation' })).toBeVisible(); }); ``` Add `// @ts-check` at the start of each test file when using JavaScript in VS Code to get automatic type checking. [writing_tests#first-test]

Current answer

You can use `context.waitForEvent('page')` to wait for and get the new page that is created in the context [pages#handling-new-pages]. Alternatively, you can listen for new pages using `context.on('page', ...)` [pages#handling-new-pages].

regressednet-wait-for-responseHow do I wait for a specific API response before asserting on the page?-0.062

case went from passing to failing

Now failing: grounding

Baseline answer

You can monitor all the Requests and Responses: ```js // Subscribe to 'request' and 'response' events. page.on('request', request => console.log('>>', request.method(), request.url())); page.on('response', response => console.log('<<', response.status(), response.url())); await page.goto('https://example.com'); ``` Or wait for a network response after the button click with page.waitForResponse(): ```js // Use a glob URL pattern. [network#network-events] Note no await. const responsePromise = page.waitForResponse('**/api/fetch_data'); await page.getByText('Update').click(); const response = await responsePromise; ```

Current answer

You can wait for a specific network response by using `page.waitForResponse()` with a glob URL pattern, regular expression, or predicate before performing an action like clicking a button [network#variations, network#network-events]. Then, await the resulting response promise before making assertions [network#variations, network#network-events].

regressedpar-retries-defaultAre failing tests retried by default?-0.723

case went from passing to failing

Now failing: expected_citation_overlap, keyword_presence, refusal_behaviour

Baseline answer

When enabled, failing tests will be retried multiple times until they pass, or until the maximum number of retries is reached. [test_retries#retries] By default failing tests are not retried. ```bash npx playwright test --retries=3 ``` You can configure retries in the configuration file: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Give failing tests 3 retry attempts retries: 3, }); ``` Playwright Test will categorize tests as follows: - "passed" - tests that passed on the first run; - "flaky" - tests that failed on the first run, but passed when retried; - "failed" - tests that failed on the first run and failed all retries. ```bash Running 3 tests using 1 worker ✓ example.spec.ts:4:2 › first passes (438ms) x example.spec.ts:5:2 › second flaky (691ms) ✓ example.spec.ts:5:2 › second flaky (522ms) ✓ example.spec.ts:6:2 › third passes (932ms) 1 flaky example.spec.ts:5:2 › second flaky 2 passed (4s) ``` You can detect retries at runtime with testInfo.retry, which is accessible to any test, hook or fixture.

Current answer

I do not have enough information in the provided documents to answer that.

regressedpar-retries-flagHow do I make Playwright retry failing tests?-0.096

case went from passing to failing

Now failing: grounding

Baseline answer

A common question when adding accessibility tests to an application is "how do I suppress known violations?" The following examples demonstrate a few techniques you can use. [accessibility_testing#handling-known-issues] By default failing tests are not retried. ```bash npx playwright test --retries=3 ``` You can configure retries in the configuration file: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Give failing tests 3 retry attempts retries: 3, }); ``` Playwright Test will categorize tests as follows: - "passed" - tests that passed on the first run; - "flaky" - tests that failed on the first run, but passed when retried; - "failed" - tests that failed on the first run and failed all retries. ```bash Running 3 tests using 1 worker ✓ example.spec.ts:4:2 › first passes (438ms) x example.spec.ts:5:2 › second flaky (691ms) ✓ example.spec.ts:5:2 › second flaky (522ms) ✓ example.spec.ts:6:2 › third passes (932ms) 1 flaky example.spec.ts:5:2 › second flaky 2 passed (4s) ``` You can detect retries at runtime with testInfo.retry, which is accessible to any test, hook or fixture. [test_retries#retries]

Current answer

You can enable test retries in Playwright by passing the `--retries` command-line flag or by configuring the `retries` option in your configuration file [test_retries#retries].

regressedpar-shard-flagHow do I split my suite across four CI machines?-0.996

case went from passing to failing

Now failing: expected_citation_overlap, keyword_presence, refusal_behaviour

Baseline answer

For example, to split the suite into four shards, each running one fourth of the tests: ```bash npx playwright test --shard=1/4 npx playwright test --shard=2/4 npx playwright test --shard=3/4 npx playwright test --shard=4/4 ``` Now, if you run these shards in parallel on different jobs, your test suite completes four times faster. [test_sharding#sharding-tests-between-multiple-machines] Playwright Test can shard a test suite, so that it can be executed on multiple machines. [test_parallel#shard-tests-between-multiple-machines]

Current answer

I do not have enough information in the provided documents to answer that.

regressedtool-install-one-browserHow do I install only WebKit instead of all browsers?-0.999

case went from passing to failing

Now failing: expected_citation_overlap, keyword_presence, refusal_behaviour

Baseline answer

See [system requirements](./intro.md#system-requirements). ```bash npm i -D playwright ``` You will also need to install browsers - either manually or by adding a package that will do it for you automatically. ```bash npx playwright install chromium firefox webkit npm i -D @playwright/browser-chromium @playwright/browser-firefox @playwright/browser-webkit ``` See [managing browsers](./browsers.md#managing-browser-binaries) for more options. [library#usage] Running the command without arguments will install the default browsers. ```bash npx playwright install ``` ```bash mvn exec:java -e -D exec.mainClass=com.microsoft.playwright.CLI -D exec.args="install" ``` ```bash playwright install ``` ```bash pwsh bin/Debug/netX/playwright.ps1 install ``` You can also install specific browsers by providing an argument: ```bash npx playwright install webkit ``` ```bash mvn exec:java -e -D exec.mainClass=com.microsoft.playwright.CLI -D exec.args="install webkit" ``` ```bash playwright install webkit ``` ```bash pwsh bin/Debug/netX/playwright.ps1 install webkit ``` See all supported browsers: ```bash npx playwright install --help ``` ```bash mvn exec:java -e -D exec.mainClass=com.microsoft.playwright.CLI -D exec.args="install --help" ``` ```bash playwright install --help ``` ```bash pwsh bin/Debug/netX/playwright.ps1 install --help ``` [browsers#install-browsers]

Current answer

I do not have enough information in the provided documents to answer that.

regressedtool-open-traceHow do I open a trace.zip file that CI produced?-0.294

case went from passing to failing

Now failing: grounding, keyword_presence

Baseline answer

Make sure to add the full path to where your `trace.zip` file is located. ```bash npx playwright show-trace path/to/trace.zip ``` ```bash mvn exec:java -e -D exec.mainClass=com.microsoft.playwright.CLI -D exec.args="show-trace trace.zip" ``` ```bash playwright show-trace trace.zip ``` ```bash pwsh bin/Debug/netX/playwright.ps1 show-trace trace.zip ``` [trace_viewer#opening-trace-viewer] This will produce a `trace.zip` file for each test that was retried. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ retries: 1, use: { trace: 'on-first-retry', }, }); ``` ```js const browser = await chromium.launch(); const context = await browser.newContext(); // Start tracing before creating / navigating a page. await context.tracing.start({ screenshots: true, snapshots: true }); const page = await context.newPage(); await page.goto('https://playwright.dev'); // Stop tracing and export it into a zip archive. await context.tracing.stop({ path: 'trace.zip' }); ``` Available options to record a trace: - `'on-first-retry'` - Record a trace only when retrying a test for the first time. - `'on-all-retries'` - Record traces for all test retries. - `'off'` - Do not record a trace. - `'on'` - Record a trace for each test. (not recommended as it's performance heavy) - `'retain-on-failure'` - Record a trace for each test, but remove it from successful test runs. [trace_viewer#tracing-on-ci]

Current answer

You can download the trace file produced on CI and open it by uploading it via drag and drop at [trace.playwright.dev](https://trace.playwright.dev) [ci_intro#viewing-the-trace-3]. Alternatively, you can open a saved trace using the Playwright CLI or directly in the browser at the same website [trace_viewer#opening-trace-viewer].

degradedcfg-fixture-orderIf fixture A depends on fixture B, which one is torn down first?-0.071

score dropped by 0.0707 (threshold 0.01) but still passing

Baseline answer

Fixtures follow these rules to determine the execution order: * When fixture A depends on fixture B: B is always set up before A and torn down after A. * Non-automatic fixtures are executed lazily, only when the test/hook needs them. * Test-scoped fixtures are torn down after each test, while worker-scoped fixtures are only torn down when the worker process executing tests is torn down. [test_fixtures#execution-order] A few observations: * `page` and `autoTestFixture` are set up and torn down for each test, as test-scoped fixtures. * `unusedFixture` is never set up because it is not used by any tests/hooks. * `testFixture` depends on `workerFixture` and triggers its setup. * `workerFixture` is lazily set up before the second test, but torn down once during worker shutdown, as a worker-scoped fixture. * `autoWorkerFixture` is set up for `beforeAll` hook, but `autoTestFixture` is not. [test_fixtures#execution-order~5]

Current answer

Fixture A is torn down first, while fixture B is torn down after A [test_fixtures#execution-order].

degradedcfg-focus-testHow do I run only one test while I'm debugging it?-0.108

score dropped by 0.1077 (threshold 0.01) but still passing

Baseline answer

You can run tests in headed mode using the `--headed` CLI argument, or run your tests in [UI mode](./test-ui-mode.md) using the `--ui` flag to see a full trace of your tests. **You will learn** - [How to run tests from the command line](/running-tests.md#command-line) - [How to debug tests](/running-tests.md#debugging-tests) - [How to open the HTML test reporter](/running-tests.md#test-reports) [running_tests#introduction] Use the Copy Locator button to copy the locator to your clipboard and then paste it into your test. ![Locator picker in the Playwright Inspector](./images/getting-started/run-tests-pick-locator.png) Check out our [debugging guide](./debug.md) to learn more about debugging with the [VS Code debugger](./debug.md#vs-code-debugger), UI Mode, and the [Playwright Inspector](./debug.md#playwright-inspector) as well as debugging with [Browser Developer tools](./debug.md#browser-developer-tools). [running_tests#debug-tests-with-the-playwright-inspector~2]

Current answer

To debug one test, run the test command followed by the test file name, the line number, and the `--debug` flag [debug#debug-one-test-on-all-browsers]. For example, use `npx playwright test example.spec.ts:10 --debug` [debug#debug-one-test-on-all-browsers].

degradedcfg-github-annotationsIs there a reporter that annotates failures directly in GitHub Actions?-0.011

score dropped by 0.0113 (threshold 0.01) but still passing

Baseline answer

You can use the built in `github` reporter to get automatic failure annotations when running in GitHub actions. [test_reporters#github-actions-annotations] Also, it is not recommended to use this annotation type if running your tests with a matrix strategy as the stack trace failures will multiply and obscure the GitHub file view. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // 'github' for GitHub Actions CI to generate annotations, plus a concise 'dot' // default 'list' when running locally reporter: process.env.CI ? 'github' : 'list', }); ``` The `github` reporter accepts `omitTags` (or the `PLAYWRIGHT_GITHUB_OMIT_TAGS` environment variable) to suppress test tags in its annotations, for example `reporter: [['github', { omitTags: true }]]`.

Current answer

Yes, you can use the built-in `github` reporter to get automatic failure annotations when running in GitHub actions [test_reporters#github-actions-annotations].

degradedcfg-tag-testsHow do I tag tests as smoke so I can run just those?-0.659

score dropped by 0.6594 (threshold 0.01) but still passing

Now failing: refusal_behaviour

Baseline answer

We can split tests into projects and use filters to run a subset of tests. [test_projects#splitting-tests-into-projects] A common question when adding accessibility tests to an application is "how do I suppress known violations?" The following examples demonstrate a few techniques you can use. [accessibility_testing#handling-known-issues]

Current answer

I do not have enough information in the provided documents to answer that.

degradedcfg-update-snapshotsThe UI changed on purpose and my screenshot comparisons fail. How do I regenerate the baselines?-0.057

score dropped by 0.0565 (threshold 0.01) but still passing

Baseline answer

Sometimes you need to update the reference screenshot, for example when the page has changed. [test_snapshots#updating-screenshots] Do this with the `--update-snapshots` flag. ```bash npx playwright test --update-snapshots ```

Current answer

You can update the reference screenshot by using the `--update-snapshots` flag [test_snapshots#updating-screenshots].

degradedcfg-webserverCan Playwright start my app's dev server before the tests instead of me running it separately?-0.072

score dropped by 0.0725 (threshold 0.01) but still passing

Baseline answer

In this case, you can increase the timeout to wait for the server to start. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Rest of your config... // Run your local dev server before starting the tests webServer: { command: 'npm run start', url: 'http://localhost:3000', reuseExistingServer: !process.env.CI, timeout: 120 * 1000, }, }); ``` [test_webserver#adding-a-server-timeout] Use the `webServer` property in your Playwright config to launch a development web server during the tests. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Run your local dev server before starting the tests webServer: { command: 'npm run start', url: 'http://localhost:3000', reuseExistingServer: !process.env.CI, stdout: 'ignore', stderr: 'pipe', }, }); ``` | Property | Description | | :- | :- | | testConfig.webServer | Launch a development web server (or multiple) during the tests. | | `command`| Shell command to start the local dev server of your app. | | `cwd` | Current working directory of the spawned process, defaults to the directory of the configuration file. | | `env` | Environment variables for the command. [test_webserver#configuring-a-web-server]

Current answer

Yes, Playwright includes a `webServer` option in its configuration file to launch a local dev server before running your tests [test_webserver#introduction].

degradedloc-css-vs-roleWhy shouldn't I select a button by its CSS class in my test?-0.072

score dropped by 0.0723 (threshold 0.01) but still passing

Baseline answer

Comma-separated list of CSS selectors will match all elements that can be selected by one of the selectors in that list. ```js // Clicks a <button> that has either a "Log in" or "Sign in" text. await page.locator('button:has-text("Log in"), button:has-text("Sign in")').click(); ``` The `:is()` pseudo-class is a [CSS pseudo-class](https://developer.mozilla.org/en-US/docs/Web/CSS/:is) that may be useful for specifying a list of extra conditions on an element. [other_locators#css-elements-matching-one-of-the-conditions] Should the designer change something then the class might change, thus breaking your test. ```js // 👎 page.locator('button.buttonIcon.episode-actions-later'); ``` Use locators that are resilient to changes in the DOM. ```js // 👍 page.getByRole('button', { name: 'submit' }); ``` [best_practices#prefer-user-facing-attributes-to-xpath-or-css-selectors]

Current answer

Your DOM can easily change, so depending on your DOM structure (such as a CSS class) can lead to failing tests if a designer changes something [best_practices#prefer-user-facing-attributes-to-xpath-or-css-selectors].

degradedloc-force-clickWhat does the force option on click actually skip?-0.075

score dropped by 0.0752 (threshold 0.01) but still passing

Baseline answer

Some actions like locator.click() support `force` option that disables non-essential actionability checks, for example passing truthy `force` to locator.click() method will not check that the target element actually receives click events. [actionability#forcing-actions] You can skip certain test based on the condition. ```js test('skip this test', async ({ page, browserName }) => { test.skip(browserName === 'firefox', 'Still working on it'); }); ``` [test_annotations#conditionally-skip-a-test]

Current answer

The `force` option disables non-essential actionability checks, such as verifying that the target element actually receives click events [actionability#forcing-actions, input#forcing-the-click].

degradedloc-strict-modeMy click threw an error saying the locator resolved to more than one element. Why does Playwright do that?-0.827

score dropped by 0.8268 (threshold 0.01) but still passing

Now failing: expected_citation_overlap, refusal_behaviour

Baseline answer

The log tells you if the element was visible, enabled and stable, if the locator resolved to an element, scrolled into view, and so much more. [debug#actionability-logs] This means that all operations on locators that imply some target DOM element will throw an exception if more than one element matches. [locators#strictness]

Current answer

I do not have enough information in the provided documents to answer that.

degradednet-dialog-defaultWhat happens to an alert() dialog if my test never handles it?-0.092

score dropped by 0.0922 (threshold 0.01) but still passing

Baseline answer

As a result, the following snippet will never resolve: WRONG! ```js page.on('dialog', dialog => console.log(dialog.message())); await page.getByRole('button').click(); // Will hang here ``` If there is no listener for the 'dialog' event, all dialogs are automatically dismissed. [dialogs#alert-confirm-prompt-dialogs] If a certain event needs to be handled once, there is a convenience API for that: ```js page.once('dialog', dialog => dialog.accept('2021')); await page.evaluate("prompt('Enter a number:')"); ``` [events#adding-one-off-listeners]

Current answer

By default, dialogs like `alert()` are automatically dismissed by Playwright if there is no listener for the 'dialog' event [dialogs#alert-confirm-prompt-dialogs].

degradednet-glob-double-starIn a page.route pattern, what is the difference between * and **?-0.667

score dropped by 0.6667 (threshold 0.01) but still passing

Now failing: refusal_behaviour

Baseline answer

Set up route on the entire browser context with browserContext.route() or page with page.route(). [network#variations-2] It will apply to popup windows and opened links. ```js await browserContext.route('**/api/login', route => route.fulfill({ status: 200, body: 'accept', })); await page.goto('https://example.com'); ```

Current answer

I do not have enough information in the provided documents to answer that.

degradednet-msw-missing-eventsI use Mock Service Worker in the app and page.route never sees the requests. What is going on?-0.033

score dropped by 0.0333 (threshold 0.01) but still passing

Baseline answer

While this tool works out of the box for mocking responses, it adds its own Service Worker that takes over the network requests, hence making them invisible to browserContext.route() and page.route(). [network#missing-network-events-and-service-workers] If you are interested in both network testing and mocking, consider using built-in browserContext.route() and page.route() for [response mocking](#handle-requests). ###### If you're interested in not solely using Service Workers for testing and network mocking, but in routing and listening for requests made by Service Workers themselves, please see [this guide](./service-workers.md).

Current answer

Mock Service Worker adds its own Service Worker that takes over network requests, making them invisible to Playwright's `page.route()` and `browserContext.route()` [network#missing-network-events-and-service-workers]. To fix this, you can disable Service Workers by setting `browser.newContext.serviceWorkers` to `'block'` [network#missing-network-events-and-service-workers].

degradedpar-isolation-contextHow does Playwright keep one test's cookies and local storage from leaking into the next test?-0.101

score dropped by 0.1014 (threshold 0.01) but still passing

Baseline answer

They depend on your application's authentication model which may require some combination of cookies, local storage, IndexedDB or passkeys. [auth#reusing-signed-in-state] This means that each test has its own local storage, session storage, cookies etc. [browser_contexts#what-is-test-isolation]

Current answer

Playwright runs tests in separate worker processes, each equipped with its own isolated BrowserContext to isolate cookies, storage, and in-memory globals [test_parallel#avoiding-shared-state-in-parallel-tests].

degradedpar-max-failuresCan I stop the whole run after ten failures so I don't waste CI minutes?-0.044

score dropped by 0.0443 (threshold 0.01) but still passing

Baseline answer

You can limit the number of failed tests in the whole test suite by setting `maxFailures` config option or passing `--max-failures` command line flag. [test_parallel#limit-failures-and-fail-fast] It is the default on CI and useful where you don't want a lot of output. ```bash npx playwright test --reporter=dot ``` ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ reporter: 'dot', }); ``` Here is an example output in the middle of a test run. [test_reporters#dot-reporter]

Current answer

Yes, you can limit the number of failed tests by setting the `maxFailures` config option or passing the `--max-failures` command line flag [test_parallel#limit-failures-and-fail-fast]. For example, you can set it to 10 to stop the test suite after reaching that number of failures [test_parallel#limit-failures-and-fail-fast].

degradedpar-serial-modeMy tests depend on each other. How do I make them run in order and stop after the first failure?-0.138

score dropped by 0.1382 (threshold 0.01) but still passing

Baseline answer

:- | :- | :- | | `'off'` | — | — | — | | `'on'` | first run | first run + retry | all three runs | | `'retain-on-failure'` | — | first run | all three runs | | `'retain-on-first-failure'` | — | first run | first run | | `'retain-on-failure-and-retries'` | — | first run + retry | all three runs | | `'on-first-retry'` | — | first retry | first retry | | `'on-all-retries'` | — | first retry | both retries | [test_use_options#video-modes~2] Trace files, screenshots and videos will appear in the test output directory, typically `test-results`. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ use: { // Capture screenshot after each test failure. screenshot: 'only-on-failure', // Record trace only when retrying a test for the first time. trace: 'on-first-retry', // Record video only when retrying a test for the first time. video: 'on-first-retry' }, }); ``` | Option | Description | | :- | :- | | testOptions.screenshot | Capture [screenshots](./screenshots.md) of your test. [test_use_options#recording-options]

Current answer

To run test files in order, you must disable parallel test execution so that Playwright Test runs them alphabetically [test_parallel#sort-test-files-alphabetically]. To stop the test suite after failures, you can set the `maxFailures` config option or pass the `--max-failures` command-line flag [test_parallel#limit-failures-and-fail-fast].

degradedpar-workers-cliHow do I limit the number of parallel workers on CI but keep the default locally?-0.051

score dropped by 0.0506 (threshold 0.01) but still passing

Baseline answer

From the command line: ```bash npx playwright test --workers 4 ``` In the configuration file: ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Limit the number of workers on CI, use default locally workers: process.env.CI ? [test_parallel#limit-workers] Useful on CI. | | `--fully-parallel` | Run all tests in parallel (default: false). | | `--global-timeout <timeout>` | Maximum time this test suite can run in milliseconds (default: unlimited). | | `-g <grep>` or `--grep <grep>` | Only run tests matching this regular expression (default: ".*"). | | `-G <grep>` or `--grep-invert <grep>` | Only run tests that do not match this regular expression. | | `--headed` | Run tests in headed browsers (default: headless). | | `--ignore-snapshots` | Ignore screenshot and snapshot expectations. | | `-j <workers>` or `--workers <workers>` | Number of concurrent workers or percentage of logical CPU cores, use 1 to run in a single worker (default: 50%). | | `--last-failed` | Only re-run the failures. | | `--last-failed-file <file>` | Override the default last-run JSON path for `--last-failed` (default: `<outputDir>/.last-run.json`). [test_cli#all-options~2]

Current answer

You can limit the number of workers on CI while using the default locally by setting `workers: process.env.CI ? 2 : undefined` (or `1` for CI) in your configuration file [test_parallel#limit-workers, ci#workers].

degradedto-beforeall-timeoutHow do I give a beforeAll hook more time than a normal test?-0.055

score dropped by 0.0548 (threshold 0.01) but still passing

Baseline answer

You can change it separately for each hook by calling testInfo.setTimeout() inside the hook. ```js import { test, expect } from '@playwright/test'; test.beforeAll(async () => { // Set timeout for this hook. test.setTimeout(60000); }); ``` API reference: testInfo.setTimeout(). [test_timeouts#change-timeout-for-beforeall-afterall-hook] A common question when adding accessibility tests to an application is "how do I suppress known violations?" The following examples demonstrate a few techniques you can use. [accessibility_testing#handling-known-issues]

Current answer

You can change the timeout for a `beforeAll` hook separately by calling `testInfo.setTimeout()` inside the hook [test_timeouts#change-timeout-for-beforeall-afterall-hook].

degradedto-fixture-timeoutCan a slow worker fixture have its own timeout instead of eating into the test timeout?-0.046

score dropped by 0.0458 (threshold 0.01) but still passing

Baseline answer

You can set a separate larger timeout for such a fixture, and keep the overall test timeout small. ```js import { test as base, expect } from '@playwright/test'; const test = base.extend<{ slowFixture: string }>({ slowFixture: [async ({}, use) => { // ... perform a slow operation ... await use('hello'); }, { timeout: 60000 }] }); test('example test', async ({ slowFixture }) => { // ... }); ``` Unlike regular test-scoped fixtures, each [worker-scoped](#worker-scoped-fixtures) fixture has its own timeout, equal to the test timeout. [test_fixtures#fixture-timeout] This way you can keep the overall test timeout small, and give the slow fixture more time. ```js import { test as base, expect } from '@playwright/test'; const test = base.extend<{ slowFixture: string }>({ slowFixture: [async ({}, use) => { // ... perform a slow operation ... await use('hello'); }, { timeout: 60_000 }] }); test('example test', async ({ slowFixture }) => { // ... }); ``` API reference: test.extend(). [test_timeouts#fixture-timeout]

Current answer

Yes, you can set a separate, larger timeout for a slow worker-scoped fixture rather than sharing the overall test timeout [test_fixtures#fixture-timeout, test_timeouts#fixture-timeout]. You can change this timeout by passing the `timeout` option to the fixture [test_fixtures#worker-scoped-fixtures].

degradedto-global-timeoutIs there a timeout for the whole test run, and what is it by default?-0.050

score dropped by 0.0503 (threshold 0.01) but still passing

Baseline answer

Playwright Test supports a timeout for the whole test run. [test_timeouts#global-timeout] Global timeout produces the following error: ```txt Running 1000 tests using 10 workers 514 skipped 486 passed Timed out waiting 3600s for the entire test run ``` You can set global timeout in the config. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ globalTimeout: 3_600_000, }); ``` API reference: testConfig.globalTimeout.

Current answer

Yes, Playwright Test supports a timeout for the whole test run, but there is no default global timeout [test_timeouts#global-timeout].

degradedto-webserver-timeoutMy dev server takes ages to boot and Playwright gives up waiting for it. What do I change?-0.085

score dropped by 0.0850 (threshold 0.01) but still passing

Baseline answer

In this case, you can increase the timeout to wait for the server to start. ```js import { defineConfig } from '@playwright/test'; export default defineConfig({ // Rest of your config... // Run your local dev server before starting the tests webServer: { command: 'npm run start', url: 'http://localhost:3000', reuseExistingServer: !process.env.CI, timeout: 120 * 1000, }, }); ``` [test_webserver#adding-a-server-timeout] Playwright comes with a `webServer` option in the config file which gives you the ability to launch a local dev server before running your tests. [test_webserver#introduction]

Current answer

You can increase the timeout by setting the `timeout` property within the `webServer` configuration in your Playwright config file [test_webserver#adding-a-server-timeout].

degradedtool-codegenHow do I record a test by clicking through the site instead of writing it by hand?-0.083

score dropped by 0.0833 (threshold 0.01) but still passing

Baseline answer

Traces can be recorded by running your tests with the `--tracing` flag. ```bash pytest --tracing on ``` Options for tracing are: - `on`: Record trace for each test - `off`: Do not record trace. (default) - `retain-on-failure`: Record trace for each test, but remove all traces from successful test runs. [trace_viewer#recording-a-trace-2] Instead of writing tests from scratch, you can simply navigate through your application while CodeGen captures your actions and converts them into reliable test code with proper locators and assertions. <LiteYouTube id="5XIZPqKkdBA" title="Generating Playwright tests in VS Code" /> - **Record a New Test**: Click **Record new** in the sidebar. [getting_started_vscode#generating-tests-with-codegen]

Current answer

To record a new test, click on the **Record new** button from the Testing sidebar, which creates a file and opens a browser window [codegen#record-a-new-test]. Then, go to the URL you wish to test and click around to record your user actions, which will be generated directly as test code in VS Code [codegen#record-a-new-test].

degradedtool-docker-imageDoes the official Docker image include the Playwright npm package itself?-0.051

score dropped by 0.0507 (threshold 0.01) but still passing

Baseline answer

Use the [Playwright Docker image](./docker.md) to run tests on Jenkins. [ci#jenkins] The Playwright package/dependency is not included in the image and should be installed separately. [docker#introduction]

Current answer

No, the Playwright package/dependency is not included in the Docker image and should be installed separately [docker#introduction].

degradedtool-ui-modeWhat is the command to open UI mode?-0.035

score dropped by 0.0347 (threshold 0.01) but still passing

Baseline answer

To open UI mode, run the following command in your terminal: ```bash npx playwright test --ui ``` [test_ui_mode#opening-ui-mode] You can also use [UI Mode](./test-ui-mode.md) for a better developer experience, as it traces each test automatically. ```bash npx playwright test --trace on ``` You can then open the HTML report and click on the trace icon to open the trace. ```bash npx playwright show-report ``` [trace_viewer#tracing-locally]

Current answer

To open UI mode, run the command `npx playwright test --ui` in your terminal [test_ui_mode#opening-ui-mode].

improvedauth-shared-accountWhat is the recommended way to log in once and reuse the session across all tests?+0.045

score rose by 0.0451 (threshold 0.01)

Now passing: expected_citation_overlap

Baseline answer

await page.goto('https://github.com/login'); await page.getByLabel('Username or email address').fill('username'); await page.getByLabel('Password').fill('password'); await page.getByRole('button', { name: 'Sign in' }).click(); }); test('first', async ({ page }) => { // page is signed in. }); test('second', async ({ page }) => { // page is signed in. }); ``` You can also reuse the signed-in state in the tests with [setup project](./auth.md#basic-shared-account-in-all-tests). [best_practices#make-tests-as-isolated-as-possible~2] That way you can log in only once and then skip the log in step for all of the tests.

Current answer

The recommended approach is to authenticate once in a setup project, save the authentication state, and then reuse it to bootstrap each test already authenticated [auth#basic-shared-account-in-all-tests]. You can also reuse the signed-in state in tests by using the setup project to log in only once and skip the log-in step for all tests [best_practices#make-tests-as-isolated-as-possible~2].

improvedcfg-auto-fixtureCan a fixture run for every test without each test having to list it as a parameter?+0.060

case went from failing to passing

Now passing: keyword_presence

Baseline answer

For example, you can create a login test that runs first, then reuse that authenticated state across multiple tests without having to log in again for each test. [getting_started_vscode#project-dependencies] person }, use) => { await page.goto('/chat'); // We use "person" parameter as a "name" for the chat room. await page.getByLabel('User Name').fill(person); await page.getByText('Enter chat room').click(); // Each test will get a "page" that already has the person name. await use(page); }, }); ``` ```js import { test as base } from '@playwright/test'; export type TestOptions = { person: string; }; export const test = base.extend<TestOptions>({ // Define an option and provide a default value. // We can later override it in the config. person: ['John', { option: true }], // Override default "page" fixture. page: async ({ page, person }, use) => { await page.goto('/chat'); // We use "person" parameter as a "name" for the chat room. await page.getByLabel('User Name').fill(person); await page.getByText('Enter chat room').click(); // Each test will get a "page" that already has the person name. await use(page); }, }); ``` Parameterized projects behavior has changed in version 1.18. [Learn more](./release-notes#breaking-change-custom-config-options). [test_parameterize#parameterized-projects~3]

Current answer

Yes, you can declare them as auto fixtures with `scope: 'worker'` and `auto: true` so they automatically run for every worker without being redeclared in every file [test_fixtures#adding-global-beforeall-afterall-hooks].

improvedcfg-run-one-projectHow do I run only the firefox project from the command line?+0.077

case went from failing to passing

Now passing: keyword_presence

Baseline answer

All test filtering options, such as `--grep`/`--grep-invert`, `--shard`, filtering directly by location in the command line, or using [`test.only()`](./api/class-test.md#test-only), directly select the primary tests to be run. [test_global_setup_teardown#test-filtering] All test filtering options, such as `--grep`/`--grep-invert`, `--shard`, filtering directly by location in the command line, or using [`test.only()`](./api/class-test.md#test-only), directly select the primary tests to be run. [test_projects#test-filtering]

Current answer

To run tests on the Firefox project, use the `--project` flag followed by `firefox` [running_tests#run-tests-on-different-browsers].

improvedloc-actionability-checksWhat checks does Playwright do automatically before clicking an element?+0.147

score rose by 0.1473 (threshold 0.01)

Now failing: grounding

Now passing: expected_citation_overlap, keyword_presence

Baseline answer

Auto waiting means that Playwright performs a range of actionability checks on the elements, such as ensuring the element is visible and enabled before it performs the click. [best_practices#use-locators] This can help you understand what happened during your test and what Playwright did or tried to do. [debug#actionability-logs]

Current answer

Before clicking an element, Playwright ensures that the locator resolves to exactly one element, and that the element is visible, stable, receives events, and is enabled [actionability#introduction]. Additionally, locators perform auto-waiting to check that elements are visible and enabled before executing a click [best_practices#use-locators].

improvedloc-testid-tradeoffWhat is the downside of locating elements by test id?+0.265

case went from failing to passing

Now passing: expected_citation_overlap, keyword_presence

Baseline answer

We recommend [locating by test id](./locators.md#locate-by-test-id) instead. [other_locators#id-data-testid-data-test-id-data-test-selectors] Currently, only the following attributes are supported: - `id` - `data-testid` - `data-test-id` - `data-test` ```js // Fill an input with the id "username" await page.locator('id=username').fill('value'); // Click an element with data-test-id "submit" await page.locator('data-test-id=submit').click(); ``` Attribute selectors are not CSS selectors, so anything CSS-specific like `:enabled` is not supported.

Current answer

The downside of locating elements by test id is that test ids are not user-facing [locators#locate-by-test-id].

improvednet-abort-imagesCan I block all image requests to make tests faster?+0.091

score rose by 0.0910 (threshold 0.01)

Now passing: expected_citation_overlap

Baseline answer

Playwright Test reuses a single worker as much as it can to make testing faster, so multiple test files are usually run in a single worker one after another. [test_parallel#worker-processes] Just define a custom Route that mocks network for a browser context. ```js import { test, expect } from '@playwright/test'; test.beforeEach(async ({ context }) => { // Block any css requests for each test in this file. await context.route(/.css$/, route => route.abort()); }); test('loads page without css', async ({ page }) => { await page.goto('https://playwright.dev'); // ... test goes here }); ``` Alternatively, you can use page.route() to mock network in a single page. ```js import { test, expect } from '@playwright/test'; test('loads page without images', async ({ page }) => { // Block png and jpeg images. await page.route(/(png|jpeg)$/, route => route.abort()); await page.goto('https://playwright.dev'); // ... test goes here }); ``` [network#network-mocking]

Current answer

Yes, you can abort requests using `page.route()` and `route.abort()` to block image requests like png and jpeg [network#abort-requests, network#network-mocking].

improvednet-download-eventHow do I get hold of a file the page downloads during a test?+0.263

case went from failing to passing

Now passing: expected_citation_overlap, keyword_presence

Baseline answer

Take a look at the following example to see how to write a test. ```js import { test, expect } from '@playwright/test'; test('has title', async ({ page }) => { await page.goto('https://playwright.dev/'); // Expect a title "to contain" a substring. await expect(page).toHaveTitle(/Playwright/); }); test('get started link', async ({ page }) => { await page.goto('https://playwright.dev/'); // Click the get started link. await page.getByRole('link', { name: 'Get started' }).click(); // Expects page to have a heading with the name of Installation. await expect(page.getByRole('heading', { name: 'Installation' })).toBeVisible(); }); ``` Add `// @ts-check` at the start of each test file when using JavaScript in VS Code to get automatic type checking. [writing_tests#first-test] This allows you to capture all network traffic for the entire context until the context is closed. ```js test('records or updates the HAR file', async ({ page }) => { // Get the response from the HAR file await page.routeFromHAR('./hars/fruit.har', { url: '*/**/api/v1/fruits', update: true, }); // Go to the page await page.goto('https://demo.playwright.dev/api-mocking'); // Assert that the fruit is visible await expect(page.getByText('Strawberry')).toBeVisible(); }); ``` [mock#recording-a-har-file~2]

Current answer

You can obtain a downloaded file by waiting for the `download` event using `page.waitForEvent('download')` and then saving it with `download.saveAs()` [downloads#introduction]. Alternatively, you can handle the event with `page.on('download', ...)` and use `download.path()` [downloads#variations].

improvednet-redirect-handlerIf a request redirects, does my route handler get called again for the redirected request?+0.020

score rose by 0.0198 (threshold 0.01)

Now passing: fuzzy_match

Baseline answer

The handler is called once, for the original request, and the browser follows the redirect on its own. response.request() returns the last request in the chain, and request.redirectedFrom() walks it back to the one you intercepted. [network#redirects] The handler runs before the network stack has processed the request: route.continue() passes it on, route.fulfill() answers it without touching the network, and route.abort() fails it. [network#how-request-interception-works]

Current answer

No, the handler is only called once for the original request, and the browser follows the redirect on its own [network#redirects].

improvedoos-cypress-retriesHow do I configure retries in Cypress?+0.262

case went from failing to passing

Now passing: refusal_behaviour

Baseline answer

Playwright has many options to configure how your tests are run. [test_configuration#introduction] A common question when adding accessibility tests to an application is "how do I suppress known violations?" The following examples demonstrate a few techniques you can use. [accessibility_testing#handling-known-issues]

Current answer

I do not have enough information in the provided documents to answer that.

improvedoos-pricingHow much does the Playwright enterprise licence cost per seat?+0.250

case went from failing to passing

Now passing: refusal_behaviour

Baseline answer

<LiteYouTube id="54_aC-rVKHg" title="Playwright 1.45" /> [clock#related-videos] Playwright executes tests in isolated environments called [browser contexts](./browser-contexts.md). [auth#introduction]

Current answer

I do not have enough information in the provided documents to answer that.

improvedoos-python-syntaxWhat is the Python syntax for page.get_by_role?+0.255

case went from failing to passing

Now passing: refusal_behaviour

Baseline answer

For example, consider the following DOM structure. ```html <button data-testid="directions">Itinéraire</button> ``` You can locate the element by its test id: ```js await page.getByTestId('directions').click(); ``` You can also use test ids when you choose to use the test id methodology or when you can't locate by [role](#locate-by-role) or [text](#locate-by-text). [locators#locate-by-test-id] For example, you can combine page.getByRole() and page.getByTitle() to match by both role and title. ```js const button = page.getByRole('button').and(page.getByTitle('Subscribe')); ``` [locators#matching-two-locators-simultaneously]

Current answer

I do not have enough information in the provided documents to answer that.

improvedoos-selenium-grid-javaWrite me a Java Selenium test that logs in to my app.+0.254

case went from failing to passing

Now passing: refusal_behaviour

Baseline answer

Playwright comes with [UI Mode](./test-ui-mode.md), where you can easily walk through each step of the test, see logs, errors, network requests, inspect the DOM snapshot, and more. [running_tests#debugging-tests] Playwright comes with a range of tooling to help you write tests. - The [VS Code extension](./getting-started-vscode.md) gives you a great developer experience when writing, running, and debugging tests. - The [test generator](./codegen.md) can generate tests and pick locators for you. - The [trace viewer](./trace-viewer.md) gives you a full trace of your tests as a local PWA that can easily be shared. [best_practices#use-playwright-s-tooling]

Current answer

I do not have enough information in the provided documents to answer that.

improvedpar-disable-parallelHow do I turn parallelism off completely?+0.272

case went from failing to passing

Now passing: expected_citation_overlap, keyword_presence

Baseline answer

A common question when adding accessibility tests to an application is "how do I suppress known violations?" The following examples demonstrate a few techniques you can use. [accessibility_testing#handling-known-issues] The following example demonstrates how to run Playwright with a CircleCI Parallelism of 4 by adding 1 to the `CIRCLE_NODE_INDEX` to pass into the `--shard` cli arg. ```yml playwright-job-name: executor: pw-noble-development parallelism: 4 steps: - run: SHARD="$((${CIRCLE_NODE_INDEX}+1))"; npx playwright test --shard=${SHARD}/${CIRCLE_NODE_TOTAL} ``` [ci#sharding-in-circleci]

Current answer

You can disable parallelism completely by setting the `workers: 1` option in the configuration file or by passing `--workers=1` to the command line [test_parallel#disable-parallelism].

improvedpar-flaky-definitionWhat does Playwright count as a "flaky" test in its results?+0.263

case went from failing to passing

Now passing: expected_citation_overlap, keyword_presence

Baseline answer

You can assert locators in order to count the items in a list. [locators#count-items-in-a-list] For example, consider the following DOM structure: ```html <ul> <li>apple</li> <li>banana</li> <li>orange</li> </ul> ``` Use the count assertion to ensure that the list has 3 items. ```js await expect(page.getByRole('listitem')).toHaveCount(3); ```

Current answer

Playwright categorizes a test as "flaky" if it failed on the first run, but passed when retried [test_retries#retries].

improvedpar-merge-reportsAfter sharding, how do I get one HTML report instead of four?+0.190

case went from failing to passing

Now passing: expected_citation_overlap, keyword_presence

Baseline answer

Check out our [sharding doc](./test-sharding) to learn more about sharding and to see a [GitHub actions example](./test-sharding.md#github-actions-example) of how to configure a job to run your tests on multiple machines as well as how to merge the HTML reports. [ci#on-push-pull-request-sharded] Get started by installing Playwright using one of the following methods. [intro#installing-playwright]

Current answer

To get one HTML report, put the blob report files from all shards into a single directory and run `npx playwright merge-reports --reporter html ./all-blob-reports` [test_sharding#merging-reports-from-multiple-shards].

improvedpar-worker-indexHow can a test find out which worker it is running in, so each worker uses its own test account?+0.154

case went from failing to passing

Now passing: expected_citation_overlap, keyword_presence

Baseline answer

Similar to how test fixtures are set up for individual test runs, worker fixtures are set up for each worker process. [test_fixtures#worker-scoped-fixtures] Below we'll create an `account` fixture that will be shared by all tests in the same worker, and override the `page` fixture to log in to this account for each test.

Current answer

To differentiate between workers and generate unique accounts, tests and fixtures can use `testInfo.workerIndex` or `workerInfo.workerIndex` [test_parallel#isolate-test-data-between-parallel-workers, test_fixtures#worker-scoped-fixtures]. Additionally, `test.info().parallelIndex` can be used as a unique identifier for a worker [auth#authenticate-with-api-request~2].

improvedto-default-test-timeoutWhat is the default timeout for a single test?+0.140

case went from failing to passing

Now passing: expected_citation_overlap, keyword_presence

Baseline answer

```js import { defineConfig } from '@playwright/test'; export default defineConfig({ timeout: 120_000, }); ``` API reference: testConfig.timeout. [test_timeouts#set-test-timeout-in-the-config] `beforeAll` and `afterAll` hooks have a separate timeout, by default equal to test timeout. [test_timeouts#change-timeout-for-beforeall-afterall-hook]

Current answer

The default timeout for each test is 30 seconds [test_timeouts#test-timeout].

improvedto-expect-timeoutHow long does expect(locator).toHaveText() keep retrying before it fails?+0.088

case went from failing to passing

Now passing: keyword_presence

Baseline answer

It produces the following error: ```txt example.spec.ts:3:1 › basic test =========================== Error: expect(received).toHaveText(expected) Expected string: "my text" Received string: "" Call log: - expect.toHaveText with timeout 5000ms - waiting for "locator('button')" ``` [test_timeouts#expect-timeout] ```js import { test, expect } from '@playwright/test'; test('example', async ({ page }) => { await expect(locator).toHaveText('hello', { timeout: 10_000 }); }); ``` [test_timeouts#specify-expect-timeout-for-a-single-assertion]

Current answer

`locatorAssertions.toHaveText()` has a default timeout of 5 seconds [test_timeouts#expect-timeout].

improvedtool-trace-on-ciWhat trace setting do the docs recommend for CI?+0.039

score rose by 0.0391 (threshold 0.01)

Now failing: grounding

Now passing: keyword_presence

Baseline answer

- [How to set up GitHub Actions](/ci-intro.md#setting-up-github-actions) - [How to view test logs](/ci-intro.md#viewing-test-logs) - [How to view the trace](/ci-intro.md#viewing-the-trace) [ci_intro#you-will-learn-2] - [How to set up GitHub Actions](/ci-intro.md#setting-up-github-actions) - [How to view test logs](/ci-intro.md#viewing-test-logs) - [How to view the HTML report](/ci-intro.md#viewing-the-html-report) - [How to view the trace](/ci-intro.md#viewing-the-trace) - [How to publish report on the web](/ci-intro.md#publishing-report-on-the-web) [ci_intro#you-will-learn]

Current answer

The documentation recommends only recording traces for failing tests [ci_intro#viewing-the-trace-3], specifically using the `on-first-retry` setting [trace_viewer_intro#recording-a-trace].

flatnet-offline-webrtcDoes the offline option also cut WebRTC connections?-0.003

no material change

Baseline answer

It does not affect WebRTC traffic: established `RTCPeerConnection`s keep sending and receiving media over UDP. [emulation#offline] It does wait until it accepts connections. [test_webserver#configuring-a-web-server~2]

Current answer

No, offline emulation does not affect WebRTC traffic, as established `RTCPeerConnection`s will keep sending and receiving media over UDP [emulation#offline].

Playwright docs - live model vs stub baseline

Summary

Retrieval metrics

By category

Cases

Regression diff

Measure	Baseline	Current	Delta
Pass rate	61.3%	53.2%	-0.081
Mean score	0.845	0.730	-0.114
Passed	38	33	-5.000
Failed	24	29	+5.000