PrivateCoffee/element-web
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 3

Improve usage of Playwright and documentation (#11999)

Browse source 
Co-authored-by: Andy Balaam <andy.balaam@matrix.org>
This commit is contained in:
Michael Telatynski 2023-12-06 10:13:48 +00:00 committed by GitHub
parent 36c6afd53f
commit d5abde0e6e
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
5 changed files with 238 additions and 16 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

15
.github/workflows/end-to-end-tests.yaml vendored
View file

@ -51,9 +51,6 @@ jobs:
# Run 4 instances in Parallel
runner: [1, 2, 3, 4]
steps:
- uses: browser-actions/setup-chrome@803ef6dfb4fdf22089c9563225d95e4a515820a0 # v1
- run: echo "BROWSER_PATH=$(which chrome)" >> $GITHUB_ENV
# There's a 'download artifact' action, but it hasn't been updated for the workflow_run action
# (https://github.com/actions/download-artifact/issues/60) so instead we get this mess:
- name: 📥 Download artifact
@ -95,7 +92,19 @@ jobs:
working-directory: matrix-react-sdk
run: yarn install --frozen-lockfile
- name: Get installed Playwright version
run: echo "PLAYWRIGHT_VERSION=$(yarn info @playwright/test -A --json | jq -r .data.version)" >> $GITHUB_ENV
- name: Cache playwright binaries
uses: actions/cache@v3
id: playwright-cache
with:
path: |
~/.cache/ms-playwright
key: ${{ runner.os }}-playwright-${{ env.PLAYWRIGHT_VERSION }}
- name: Install Playwright browsers
if: steps.playwright-cache.outputs.cache-hit != 'true'
working-directory: matrix-react-sdk
run: yarn playwright install --with-deps

4
docs/cypress.md
View file

@ -1,5 +1,9 @@
# Cypress in Element Web
# 🚨 We are moving away from Cypress in favour of Playwright.
Please do not write any new tests in Cypress and check out [the Playwright docs](playwright.md).
## Contents
- How to run the tests

219
docs/playwright.md Normal file
View file

@ -0,0 +1,219 @@
# Playwright in Element Web
## Contents
- How to run the tests
- How the tests work
- How to write great Playwright tests
- Visual testing
## Running the Tests
Our Playwright tests run automatically as part of our CI along with our other tests,
on every pull request and on every merge to develop & master.
You may need to follow instructions to set up your development environment for running
Playwright by following <https://playwright.dev/docs/browsers#install-browsers> and
<https://playwright.dev/docs/browsers#install-system-dependencies>.
However the Playwright tests are run, an element-web instance must be running on
http://localhost:8080 (this is configured in `playwright.config.ts`) - this is what will
be tested. When running Playwright tests yourself, the standard `yarn start` from the
element-web project is fine: leave it running it a different terminal as you would
when developing. Alternatively if you followed the development set up from element-web then
Playwright will be capable of running the webserver on its own if it isn't already running.
The tests use Docker to launch Homeserver (Synapse or Dendrite) instances to test against, so you'll also
need to have Docker installed and working in order to run the Playwright tests.
There are a few different ways to run the tests yourself. The simplest is to run:
```shell
docker pull matrixdotorg/synapse:develop
yarn run test:playwright
```
This will run the Playwright tests once, non-interactively.
Note: you don't need to run the `docker pull` command every time, but you should
do it regularly to ensure you are running against an up-to-date Synapse.
You can also run individual tests this way too, as you'd expect:
```shell
yarn run test:playwright --spec playwright/e2e/register/register.spec.ts
```
Playwright also has its own UI that you can use to run and debug the tests.
To launch it:
```shell
yarn run test:playwright:open --headed --debug
```
See more command line options at <https://playwright.dev/docs/test-cli>.
### Running with Rust cryptography
`matrix-js-sdk` is currently in the
[process](https://github.com/vector-im/element-web/issues/21972) of being
updated to replace its end-to-end encryption implementation to use the [Matrix
Rust SDK](https://github.com/matrix-org/matrix-rust-sdk). This is not currently
enabled by default, but it is possible to have Playwright configure Element to use
the Rust crypto implementation by passing `--project="Rust Crypto"` or using
the top left options in open mode.
## How the Tests Work
Everything Playwright-related lives in the `playwright/` subdirectory of react-sdk
as is typical for Playwright tests. Likewise, tests live in `playwright/e2e`.
`playwright/plugins/homeservers` contains Playwright plugins that starts instances
of Synapse/Dendrite in Docker containers. These servers are what Element-web runs
against in the tests.
Synapse can be launched with different configurations in order to test element
in different configurations. `playwright/plugins/homeserver/synapse/templates`
contains template configuration files for each different configuration.
Each test suite can then launch whatever Synapse instances it needs in whatever
configurations.
Note that although tests should stop the Homeserver instances after running and the
plugin also stop any remaining instances after all tests have run, it is possible
to be left with some stray containers if, for example, you terminate a test such
that the `after()` does not run and also exit Playwright uncleanly. All the containers
it starts are prefixed, so they are easy to recognise. They can be removed safely.
After each test run, logs from the Synapse instances are saved in `playwright/synapselogs`
with each instance in a separate directory named after its ID. These logs are removed
at the start of each test run.
## Writing Tests
Mostly this is the same advice as for writing any other Playwright test: the Playwright
docs are well worth a read if you're not already familiar with Playwright testing, eg.
https://playwright.dev/docs/best-practices. To avoid your tests being flaky it is also
recommended to use [auto-retrying assertions](https://playwright.dev/docs/test-assertions#auto-retrying-assertions).
### Getting a Synapse
We heavily leverage the magic of [Playwright fixtures](https://playwright.dev/docs/test-fixtures).
To acquire a homeserver within a test just add the `homeserver` fixture to the test:
```typescript
test("should do something", async ({ homeserver }) => {
// homeserver is a Synapse/Dendrite instance
});
```
This returns an object with information about the Homeserver instance, including what port
it was started on and the ID that needs to be passed to shut it down again. It also
returns the registration shared secret (`registrationSecret`) that can be used to
register users via the REST API. The Homeserver has been ensured ready to go by awaiting
its internal health-check.
Homeserver instances should be reasonably cheap to start (you may see the first one take a
while as it pulls the Docker image).
You do not need to explicitly clean up the instance as it will be cleaned up by the fixture.
### Synapse Config Templates
When a Synapse instance is started, it's given a config generated from one of the config
templates in `playwright/plugins/homeserver/synapse/templates`. There are a couple of special files
in these templates:
- `homeserver.yaml`:
Template substitution happens in this file. Template variables are:
- `REGISTRATION_SECRET`: The secret used to register users via the REST API.
- `MACAROON_SECRET_KEY`: Generated each time for security
- `FORM_SECRET`: Generated each time for security
- `PUBLIC_BASEURL`: The localhost url + port combination the synapse is accessible at
- `localhost.signing.key`: A signing key is auto-generated and saved to this file.
Config templates should not contain a signing key and instead assume that one will exist
in this file.
All other files in the template are copied recursively to `/data/`, so the file `foo.html`
in a template can be referenced in the config as `/data/foo.html`.
### Logging In
We again heavily leverage the magic of [Playwright fixtures](https://playwright.dev/docs/test-fixtures).
To acquire a logged-in user within a test just add the `user` fixture to the test:
```typescript
test("should do something", async ({ user }) => {
// user is a logged in user
});
```
You can specify a display name for the user via `test.use` `displayName`,
otherwise a random one will be generated.
This will register a random userId using the registrationSecret with a random password
and the given display name. The user fixture will contain details about the credentials for if
they are needed for User-Interactive Auth or similar but localStorage will already be seeded with them
and the app loaded (path `/`).
### Joining a Room
Many tests will also want to start with the client in a room, ready to send & receive messages. Best
way to do this may be to get an access token for the user and use this to create a room with the REST
API before logging the user in.
You can make use of the bot fixture and the `client` field on the app fixture to do this.
### Try to write tests from the users' perspective
Like for instance a user will not look for a button by querying a CSS selector.
Instead, you should work with roles / labels etc, see https://playwright.dev/docs/locators.
### Using matrix-js-sdk
Due to the way we run the Playwright tests in CI, at this time you can only use the matrix-js-sdk module
exposed on `window.matrixcs`. This has the limitation that it is only accessible with the app loaded.
This may be revisited in the future.
## Good Test Hygiene
This section mostly summarises general good Playwright testing practice, and should not be news to anyone
already familiar with Playwright.
1. Test a well-isolated unit of functionality. The more specific, the easier it will be to tell what's
wrong when they fail.
1. Don't depend on state from other tests: any given test should be able to run in isolation.
1. Try to avoid driving the UI for anything other than the UI you're trying to test. e.g. if you're
testing that the user can send a reaction to a message, it's best to send a message using a REST
API, then react to it using the UI, rather than using the element-web UI to send the message.
1. Avoid explicit waits. Playwright locators & assertions will implicitly wait for the specified
element to appear and all assertions are retried until they either pass or time out, so you should
never need to manually wait for an element.
- For example, for asserting about editing an already-edited message, you can't wait for the
'edited' element to appear as there was already one there, but you can assert that the body
of the message is what is should be after the second edit and this assertion will pass once
it becomes true. You can then assert that the 'edited' element is still in the DOM.
- You can also wait for other things like network requests in the
browser to complete (https://playwright.dev/docs/api/class-page#page-wait-for-response).
Needing to wait for things can also be because of race conditions in the app itself, which ideally
shouldn't be there!
This is a small selection - the Playwright best practices guide, linked above, has more good advice, and we
should generally try to adhere to them.
## Screenshot testing
When we previously used Cypress we also dabbled with Percy, and whilst powerful it did not
lend itself well to being executed on all PRs without needing to budget it substantially.
Playwright has built-in support for [visual comparison testing](https://playwright.dev/docs/test-snapshots).
Screenshots are saved in `playwright/snapshots` and are rendered in a Linux Docker environment for stability.
One must be careful to exclude any dynamic content from the screenshot, such as timestamps, avatars, etc,
via the `mask` option. See the [Playwright docs](https://playwright.dev/docs/test-snapshots#masking).
Some UI elements render differently between test runs, such as BaseAvatar when
there is no avatar set, choosing a colour from the theme palette based on the
hash of the user/room's Matrix ID. To avoid this creating flaky tests we inject
some custom CSS, for this to happen we use the custom assertion `toMatchScreenshot`
instead of the native `toHaveScreenshot`.
If you are running Linux and are unfortunate that the screenshots are not rendering identically,
you may wish to specify `--ignore-snapshots` and rely on Docker to render them for you.

3
package.json
View file

@ -55,6 +55,9 @@
"test:cypress:open": "cypress open",
"test:playwright": "playwright test",
"test:playwright:open": "yarn test:playwright --ui",
"test:playwright:screenshots": "yarn test:playwright:screenshots:build && yarn test:playwright:screenshots:run",
"test:playwright:screenshots:build": "docker build playwright -t matrix-react-sdk-playwright",
"test:playwright:screenshots:run": "docker run --rm --network host -v $(pwd)/../:/work/ -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/:/tmp/ -it matrix-react-sdk-playwright",
"coverage": "yarn test --coverage",
"lint:workflows": "find .github/workflows -type f \\( -iname '*.yaml' -o -iname '*.yml' \\) | xargs -I {} sh -c 'echo \"Linting {}\"; action-validator \"{}\"'"
},

13
playwright/README.md
View file

@ -1,13 +0,0 @@
To update snapshots you will need to run Playwright on a Linux machine.
If you have access to docker then you can use the following:
```shell
docker build playwright -t matrix-react-sdk-playwright
docker run \
--rm \
--network host \
-v $(pwd)/../:/work/ \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /tmp/:/tmp/ \
-it matrix-react-sdk-playwright
```