This docs describes Webdriver testing within the app.
Webdriver testing can be tricky because you're sending commands to an actual browser running on either you machine or browserstack. This can be slow but it's currently the only way to test things within a wide range of browsers without emulation. This give us the best chance of having a stable app across a range of browsers without excessive manual testing
> **A note on stability**: Webdriver tests are a lot more flakey than other types of testing, the major benefit is that you can run them on real devices, so we can hopefully get a good smoke test of various real devices. You can also screenshot those devices during test runs, to check look. You however probably don't want to write too many webdriver tests, they are best placed for smoke testing and testing stuff that's very browser specific.
**Note**: You'll need to set `BROWSERSTACK_USER` and `BROWSERSTACK_KEY` in your environment, which you can grab from <https://automate.browserstack.com/dashboard>
-`runtime` — the tldraw `App` instance in the browser window. This calls methods in the JS runtime of the app
-`ui` — methods to interacting with the apps UI elements
-`util` — some general helper methods
The `ui` is further broken up into
tldraw room for above is at <https://www.tldraw.com/v/CZw_c_7vMVkHpMcMWcMm1z8ZpN>
## Nightly tests
We run all the tests on all browsers via browserstack each night at 02:00am. You can search https://github.com/tldraw/tldraw-lite/actions/workflows/webdriver-nightly.yml for the results of the previous evenings tests
## On demand tests
To run tests on demand via github actions you can head to https://github.com/tldraw/tldraw-lite/actions/workflows/webdriver-on-demand.yml and select **run workflow** with you desired options and press the **run workflow** button
The missing tests (`.todo`) follow a lot of the same patterns that already exist in the codebase. Firefox is skipped quite a lot in the test runners, this is due to issues with the app and tests, so those also need to be resolved. You can search for `FIXME` in the `./e2e` directory to find those.
## Test writing guide
Below explains the process of writing a new test. Most actions don't have anything specific for a particular browser, however most actions do have specifics per-layout and viewport size. Lets walk through and explain a very simple test, the draw shortcut.
When pressing `d` in the browser on a desktop environment we should be changing the current state to `draw.idle`. Below is the test comments in line.
```js
// These are helpers that we use in our tests to interact with our app via the browser.
import { runtime, ui } from '../helpers'
// The runner uses mocha, `env` is an addition added by mocha-ext and `it` is
// extended to support skipping of tests in certain environments.
import { describe, env, it } from '../mocha-ext'
// Always group tests, this make them easy to run in isolation by changing
// `describe(...)` to `describe.only(...)`
describe('shortcuts', () => {
// The `env` command is used to mark groups of tests as skippable under
// certain conditions, here we're only running these tests under desktop
// environments. Note that is **REALLY** important we don't just wrap this
// in a `if(FOO) {...}` block. Otherwise `.only` would break under certain
// circumstances
env({ device: 'desktop' }, () => {
// We haven't written this test yet, we can mark those with a `.todo`
// This appears as `[TODO] {test name}` in the test output.
it.todo('select — V', async () => {
await ui.app.setup()
await browser.keys(['v'])
})
// Here is our draw shortcut test, all tests are going to be async as
// commands are sent to the browser over HTTP
it('draw — D', async () => {
// We must always `setup` our app. This starts the browser (if it
// isn't already) and does a hard reset. Note **NEVER** do this in
// a mocha `before(...)` hook. Else you'll end up starting the browser
// (doing heavy work) whether or not you actually run the test suite.
// I believe this is a mocha/webdriver integration bug.
await ui.app.setup()
// `browser` is a global from <https://webdriver.io/docs/api/browser>
// This is the default way to interact with the browser. There are
// also lots of helpers in `import { ui } from '../helpers'` module.
// Here we're instructing the browser to enter `d` to the currently
// selected element
await browser.keys(['d'])
// Although we've pressed `d` we don't know how long that's going to
// take to process in the browser. A slow environment might take
// sometime to update the runtime. So we use `waitUntil` to wait
// for the change. **DON'T** use `util.sleep(...)` here as this will
// only work if the browser/env is fast and will generally slow down
// the test runner and make things flakey.
await browser.waitUntil(async () => {
// `runtime` executes JS in the JS-runtime on the device. This
// is to test for conditions to be met.
return await runtime.isIn('draw.idle')
})
})
```
To run the above tests, we'd probably want to first isolate the test and `only` run a single test, or test group. Change
```diff
- it('draw — D', async () => {
+ it.only('draw — D', async () => {
```
Now lets start the dev server and run the tests locally. In one terminal session run
This will start a tunnel from browserstack to your local machine, running the tests against the local server. You can head to browserstack to see the completed test suite <https://automate.browserstack.com/dashboard/v2/builds/2d87ffb21f5466b93e6ef8b4007df995d23da7b3>
Spec Files: 9 passed, 9 total (100% completed) in 00:03:51
```
Now you can remove the `.only` and open a PR and rejoice in your new found skills.
```diff
- it.only('draw — D', async () => {
+ it('draw — D', async () => {
```
Existing tests are a good guide for writing more advance tests, hopefully this should give you a start 🤞
## Notes
### `msedgedriver`
You might notice that `msedgedriver` on version 91, and hasn't been updated in a while.
```
"msedgedriver": "^91.0.0",
```
This module isn't actually used but is required for `wdio-edgedriver-service` to start where we pass a custom path via the `edgedriverCustomPath` option.
### `safaridriver`
Locally safari webdriver tests are currently somewhat buggy, there appear to be some tests that don't complete. Please take on this task if you have time 🙂
You can now also run _linux_ tests on _macos_. To do this start up a selenium grid with. Note, you **must** have `docker` installed locally.
```sh
yarn e2e selenium:grid
```
Then run
```sh
yarn e2e test:local --os linux -b firefox
```
**Note**: Currently only `firefox` is supported. You can hit <http://localhost:7900/?autoconnect=1&resize=scale&password=secret> to see a VNC of the app runing in a docker container.
