Testing an iTwin Studio App with Playwright

In this guide, we’ll see how to add end to end testing capability to your iTwin Studio app using Playwright.

End-to-End (E2E) testing is a software testing method that evaluates an entire application from start to finish. It simulates real-world user scenarios to ensure that all components (frontend, backend, database, APIs, etc.) work together correctly.

Playwright is an open-source automation library developed by Microsoft for end-to-end testing of web applications.

Prerequisites

• iTwin Studio
• Enabled developer support
• Node.js v20
• pnpm
• Ensure that you are authenticated for Azure Artifacts
• A test user account credentials.

Apps created from templates

All iTwin Studio app templates come with basic Playwright tests. As such, app developers can simply add additional tests as they add extra functionality to the app.

Prerequisites

Before you can run the tests that come with the app template, you need to prepare the test environment:

• Prepare a test user account that can be used for automated testing
• For most setups, the test user account should have access to at least one iTwin and iModel (unless your app uses neither of those)
• Create an e2e/.env.production.local file
• Add the following variables:

STUDIO_TEST_USER_EMAIL_PROD=<test account email>
STUDIO_TEST_USER_PASSWORD_PROD=<test account password>
ITWIN_NAME="Project display name that the test account has access to"
IMODEL_NAME="iModel display name. iModel should belong to the iTwin specified above"

Running tests in QA environment

To run E2E tests in QA environment, create an e2e/.env.development.local file and add the following content:

STUDIO_TEST_USER_EMAIL=<test account email>
STUDIO_TEST_USER_PASSWORD=<test account password>
ITWIN_NAME="Project display name that the test account has access to"
IMODEL_NAME="iModel display name. iModel should belong to the iTwin specified above"

iTwin Studio API keys

iTwin Studio does not look up its API keys from Product Settings Service (PSS) when running in end-to-end test mode. This means services like Launch Darkly, Cesium Ion, Bing Maps, Application Insights and Amplitude are not initialized.

If you need these services you can force iTwin Studio to fetch them by setting STUDIO_E2E_FETCH_PSS_KEYS=1.

await ITwinStudioExe.create({
  ...launchOptions,
  env: {
    STUDIO_E2E_FETCH_PSS_KEYS: "1",
  },
});

E2E tests in CI

To enable E2E test running in CI, you will also need to install iTwin Studio and enable development support first.

This means that the CI runner/agent pool will need to be able to elevate permissions without password prompts.

NOTE: GitHub-hosted runners and most ADO agent pools provide this ability without additional configuration. Contact your pool owner if that ability is not available.

To install latest available iTwin Studio Beta, studio-cli provides the following command:

npx --registry="https://pkgs.dev.azure.com/bentleycs/_packaging/Packages/npm/registry/" @bentley/studio-cli@latest helpers install-latest-studio

To seamlessly enable developer support in CI environments, studio-cli provides the following command tailored specifically to run in GitHub and ADO CI:

npx --registry="https://pkgs.dev.azure.com/bentleycs/_packaging/Packages/npm/registry/" @bentley/studio-cli@latest helpers enable-dev-support-ci

This command should be invoked before running E2E tests.

After running the tests, you can clean up the CI environment by using the following command:

npx --registry="https://pkgs.dev.azure.com/bentleycs/_packaging/Packages/npm/registry/" @bentley/studio-cli@latest helpers uninstall-studio

Your pipeline might look something like this:

- script: |
        npx --registry="https://pkgs.dev.azure.com/bentleycs/_packaging/Packages/npm/registry/" @bentley/studio-cli@latest helpers install-latest-studio
      displayName: Install latest iTwin Studio
- script: |
        npx --registry="https://pkgs.dev.azure.com/bentleycs/_packaging/Packages/npm/registry/" @bentley/studio-cli@latest helpers enable-dev-support-ci
      displayName: Enable development support
- script: pnpm test-e2e
      displayName: integration tests
# only needed if the build agent is self-hosted, e.g. Mac pools on ADO
- script: |
        npx --registry="https://pkgs.dev.azure.com/bentleycs/_packaging/Packages/npm/registry/" @bentley/studio-cli@latest helpers uninstall-studio
      displayName: Cleanup - uninstall iTwin Studio
      condition: always()

FAQ

How do I find various selectors for E2E tests?

@bentley/studio-apps-testing package provides an StudioPage class that allows consuming applications reuse various iTwin Studio specific HTML selectors. This eliminates the need to find and maintain various data-test-id’s/id’s/strings that may change over time.

For example, if your E2E test requires selecting an iTwin, you could do so with the following code snippet:

import { StudioPage } from "@bentley/studio-apps-testing";
///....
app = await ITwinStudioExe.create({
  ...StubLaunchOpts,
  authConfig: TestEnvironment.authConfig,
  temporaryDirPath: tempDirectory.path,
});
studioPage = new StudioPage(app.window);

// search for the specific iTwin
await studioPage.iTwinSelection.searchBar.fill(iTwinName);
// wait for results
await app.window.getByText(iTwinName).waitFor({ state: "attached" });
await expect(studioPage.iTwinSelection.tiles.self).toHaveCount(1);
// click on the iTwin tile
await studioPage.iTwinSelection.tiles.select.first().click();
// continue with the test, e.g. wait for iModel selection page to be open
await expect(iModelSelectionPage).toBeVisible();

StudioPage class can also be extended as needed to contain selectors that are specific to your app.