New migrations again (#3220)

Describe what your pull request does. If appropriate, add GIFs or images
showing the before and after.

### Change Type

- [x] `sdk` — Changes the tldraw SDK
- [x] `galaxy brain` — Architectural changes



### Test Plan

1. Add a step-by-step description of how to test your PR here.
2.

- [ ] Unit Tests
- [ ] End to end tests

### Release Notes

#### BREAKING CHANGES

- The `Migrations` type is now called `LegacyMigrations`.
- The serialized schema format (e.g. returned by
`StoreSchema.serialize()` and `Store.getSnapshot()`) has changed. You
don't need to do anything about it unless you were reading data directly
from the schema for some reason. In which case it'd be best to avoid
that in the future! We have no plans to change the schema format again
(this time was traumatic enough) but you never know.
- `compareRecordVersions` and the `RecordVersion` type have both
disappeared. There is no replacement. These were public by mistake
anyway, so hopefully nobody had been using it.
- `compareSchemas` is a bit less useful now. Our migrations system has
become a little fuzzy to allow for simpler UX when adding/removing
custom extensions and 3rd party dependencies, and as a result we can no
longer compare serialized schemas in any rigorous manner. You can rely
on this function to return `0` if the schemas are the same. Otherwise it
will return `-1` if the schema on the right _seems_ to be newer than the
schema on the left, but it cannot guarantee that in situations where
migration sequences have been removed over time (e.g. if you remove one
of the builtin tldraw shapes).

Generally speaking, the best way to check schema compatibility now is to
call `store.schema.getMigrationsSince(persistedSchema)`. This will throw
an error if there is no upgrade path from the `persistedSchema` to the
current version.

- `defineMigrations` has been deprecated and will be removed in a future
release. For upgrade instructions see
https://tldraw.dev/docs/persistence#Updating-legacy-shape-migrations-defineMigrations

- `migrate` has been removed. Nobody should have been using this but if
you were you'll need to find an alternative. For migrating tldraw data,
you should stick to using `schema.migrateStoreSnapshot` and, if you are
building a nuanced sync engine that supports some amount of backwards
compatibility, also feel free to use `schema.migratePersistedRecord`.
- the `Migration` type has changed. If you need the old one for some
reason it has been renamed to `LegacyMigration`. It will be removed in a
future release.
- the `Migrations` type has been renamed to `LegacyMigrations` and will
be removed in a future release.
- the `SerializedSchema` type has been augmented. If you need the old
version specifically you can use `SerializedSchemaV1`

---------

Co-authored-by: Steve Ruiz <steveruizok@gmail.com>
This commit is contained in:
David Sheldrick 2024-04-15 13:53:42 +01:00 committed by GitHub
parent 63f20d1834
commit 4f70a4f4e8
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
112 changed files with 109320 additions and 106484 deletions

View file

@ -1,9 +1,75 @@
---
title: Collaboration
status: published
author: steveruizok
author: ds300
date: 3/22/2023
order: 8
---
See the [tldraw-yjs example](https://github.com/tldraw/tldraw-yjs-example) for an example of how to use yjs with the `tldraw` library.
We've designed the tldraw SDK to work with any collaboration backend. Depending on which backend you choose, you will need an interface that pipes changes coming from the editor to the backend and then merge changes from the backend back to the editor.
The best way to get started is by adapting one of our examples.
### Yjs sync example
We created a [tldraw-yjs example](https://github.com/tldraw/tldraw-yjs-example) to illustrate a way of using the [yjs](https://yjs.dev) library with the tldraw SDK. If you need a "drop in solution" for prototyping multiplayer experiences with tldraw, start here.
### Sockets example
We have a [sockets example](https://github.com/tldraw/tldraw-sockets-example) that uses [PartyKit](https://www.partykit.io/) as a backend. Unlike the yjs example, this example does not use any special data structures to handle conflicts. It should be a good starting point if you needed to write your own conflict-resolution logic.
### Our own sync engine
We developed our own sync engine for use on tldraw.com based on a push/pull/rebase-style algorithm. It powers our "shared projects", such as [this one](https://tldraw.com/r). The engine's source code can be found [here](https://github.com/tldraw/tldraw/tree/main/packages/tlsync). It was designed to be hosted on Cloudflare workers with [DurableObjects](https://developers.cloudflare.com/durable-objects/).
We don't suggest using this code directly. However, like our other examples, it may serve as a good reference for your own sync engine.
## Store data
For information about how to synchronize the store with other processes, i.e. how to get data out and put data in, including from remote sources, see the (Persistence)[/docs/persistence] page.
## User presence
Tldraw has support for displaying the 'presence' of other users. Presence information consists of:
- The user's pointer position
- The user's set of selected shapes
- The user's viewport bounds (the part of the canvas they are currently viewing)
- The user's name, id, and a color to represent them
This information will usually come from two sources:
- The tldraw editor state (e.g. pointer position, selected shapes)
- The data layer of whichever app tldraw has been embedded in (e.g. user name, user id)
Tldraw is agnostic about how this data is shared among users. However, in order for tldraw to use the presence data it needs to be put into the editor's store as `instance_presence` records.
We provide a helper for constructing a reactive signal for an `instance_presence` record locally, which can then be sent to other clients somehow. It is called [createPresenceStateDerivation](?).
```ts
import { createPresenceStateDerivation, react, atom } from 'tldraw'
// First you need to create a Signal containing the basic user details: id, name, and color
const user = atom<{ id: string; color: string; name: string }>('user', {
id: myUser.id,
color: myUser.color,
name: myUser.name,
})
// if you don't have your own user data backend, you can use our localStorage-only user preferences store
// import { getUserPreferences, computed } from 'tldraw'
// const user = computed('user', getUserPreferences)
// Then, with access to your store instance, you can create a presence signal
const userPresence = createPresenceStateDerivation(user)(store)
// Then you can listen for changes to the presence signal and send them to other clients
const unsub = react('update presence', () => {
const presence = userPresence.get()
broadcastPresence(presence)
})
```
The other clients would then call `store.put([presence])` to add the presence information to their store.
Any such `instance_presence` records tldraw finds in the store that have a different user `id` than the editor's configured user id will cause the presence information to be rendered on the canvas.

View file

@ -50,21 +50,9 @@ editor.getSelectedShapeIds() // [myShapeId, myOtherShapeId]
Each change to the state happens within a transaction. You can batch changes into a single transaction using the [Editor#batch](?) method. It's a good idea to batch wherever possible, as this reduces the overhead for persisting or distributing those changes.
### Listening for changes
### Listening for changes, and merging changes from other sources
You can subscribe to changes using the [Store#listen](?) method on [Editor#store](?). Each time a transaction completes, the editor will call the callback with a history entry. This entry contains information about the records that were added, changed, or deleted, as well as whether the change was caused by the user or from a remote change.
```ts
editor.store.listen((entry) => {
entry // { changes, source }
})
```
### Remote changes
By default, changes to the editor's store are assumed to have come from the editor itself. You can use the [Store#mergeRemoteChanges](?) method of the editor's [Editor#store](?) to make changes in the store that will be emitted via [Store#listen](?) with the `source` property as `'remote'`.
If you're setting up some kind of multiplayer backend, you would want to send only the `'user'` changes to the server and merge the changes from the server using [Store#mergeRemoteChanges](?) (`editor.store.mergeRemoteChanges`).
For information about how to synchronize the store with other processes, i.e. how to get data out and put data in, see the (Persistence)[/docs/persistence] page.
### Undo and redo

View file

@ -17,7 +17,7 @@ Persistence in tldraw means storing information about the editor's state to a da
## The `"persistenceKey"` prop
Both the `<Tldraw>` or `<TldrawEditor>` components support local persitence and cross-tab synchronization via the `persistenceKey` prop. Passing a value to this prop will persist the contents of the editor locally to the browser's IndexedDb.
Both the `<Tldraw>` or `<TldrawEditor>` components support local persistence and cross-tab synchronization via the `persistenceKey` prop. Passing a value to this prop will persist the contents of the editor locally to the browser's IndexedDb.
```tsx
import { Tldraw } from 'tldraw'
@ -54,7 +54,7 @@ export default function () {
In the example above, both editors would synchronize their document locally. They would still have two independent instance states (e.g. selections) but the document would be kept in sync and persisted under the same key.
## Snapshots
## Document Snapshots
You can get a JSON snapshot of the editor's content using the [Editor#store](?)'s [Store#getSnapshot](?) method.
@ -96,7 +96,7 @@ function LoadButton() {
A [snapshot](/reference/store/StoreSnapshot) includes both the store's [serialized records](/reference/store/SerializedStore) and its [serialized schema](/reference/store/SerializedSchema), which is used for migrations.
> By default, the `getSnapshot` method returns only the editor's document data. If you want to get records from a different scope, You can pass in `session`, `document`, `presence`, or else `all` for all scopes.
> By default, the `getSnapshot` method returns only the editor's document data. If you want to get records from a different scope, you can pass in `session`, `document`, `presence`, or else `all` for all scopes.
Note that loading a snapshot does not reset the editor's in memory state or UI state. For example, loading a snapshot during a resizing operation may lead to a crash. This is because the resizing state maintains its own cache of information about which shapes it is resizing, and its possible that those shapes may no longer exist!
@ -170,3 +170,242 @@ export default function () {
```
For a good example of this pattern, see the [yjs-example](https://github.com/tldraw/tldraw-yjs-example).
## Listening for changes
You can listen for incremental updates to the document state by calling `editor.store.listen`, e.g.
```ts
const unlisten = editor.store.listen(
(update) => {
console.log('update', update)
},
{ scope: 'document', source: 'user' }
)
```
These updates contain information about which records were added, removed, and updated. See [HistoryEntry](?)
The `scope` filter can be used to listen for changes to a specific record scope, e.g. `document`, `session`, `presence`, or `all`.
The `source` filter can be used to listen for changes from a specific source, e.g. `user`, `remote`, or `all`. (See [Store#mergeRemoteChanges](?) for more information on remote changes.)
Note that these incremental updates do not include the schema version. You should make sure that you keep a record of the latest schema version for your snapshots.
You can get the schema version by calling `editor.store.schema.serialize()` and the returned value can replace the `schema` property in the snapshot next time you need to load a snapshot. The schema does not change at runtime so you only need to do this once per session.
## Handling remote changes
If you need to synchronize changes from a remote source, e.g. a multiplayer backend, you can use the `editor.store.mergeRemoteChanges` method. This will 'tag' the changes with the `source` property as `'remote'` so you can filter them out when listening for changes.
```ts
myRemoteSource.on('change', (changes) => {
editor.store.mergeRemoteChanges(() => {
changes.forEach((change) => {
// Apply the changes to the store
editor.store.put(/* ... */)
})
})
})
```
## Migrations
Tldraw uses migrations to bring data from old snapshots up to date. These run automatically when calling `editor.store.loadSnapshot`.
### Running migrations manually
If you need to run migrations on a snapshot without loading it into the store, you can call [StoreSchema#migrateStoreSnapshot](?) directly.
```ts
import { createTLSchema } from 'tldraw'
const snapshot = await getSnapshotFromSomewhere()
const migrationResult = createTLSchema().migrateStoreSnapshot(snapshot)
if (migrationResult.type === 'success') {
console.log('Migrated snapshot', migrationResult.value)
} else {
console.error('Migration failed', migrationResult.reason)
}
```
### Custom migrations
Tldraw supports a couple of ways of adding custom data types to the tldraw store:
- [Custom shape types](/docs/shapes#Custom-shapes-1)
- [`meta` properties](/docs/shapes#Meta-information) on all of our built-in record types.
You might wish to migrate your custom data types over time as you make changes to them.
To enable this, tldraw provides two ways to add custom migrations:
1. **Shape props migrations**, specifically for migrating the shape.props objects on your custom shape types.
2. **The `migrations` config option**, which is more general purpose but much less commonly needed. This will allow you to migrate any data in the store.
#### Shape props migrations
If you have a custom shape type, you can define a `migrations` property on the shape util class. Use the `createShapePropsMigrationSequence` helper to define this property.
```ts
import { createShapePropsMigrationSequence, createShapePropsMigrationIds, ShapeUtil } from 'tldraw'
// Migrations must start a 1 and be sequential integers.
const Versions = createShapePropMigrationIds('custom-shape', {
AddColor: 1,
})
class MyCustomShapeUtil extends ShapeUtil {
static type = 'custom-shape'
static migrations = createShapePropsMigrationSequence({
sequence: [
{
id: Versions.AddColor,
up(props) {
// set the default color
props.color = 'black'
},
},
],
})
// ...
}
```
#### The `migrations` config option
First create a set of migration ids.
```ts
import { createMigrationIds } from 'tldraw'
// The first argument is a unique namespace for your migration sequence.
// We recommend using a reverse domain name, e.g. we use 'com.tldraw.foo.bar'
const SEQUENCE_ID = 'com.example.my-app'
const Versions = createMigrationIds(SEQUENCE_ID, {
// Migrations must start at 1 and be sequential integers.
AddColor: 1,
})
```
Then create a migration sequence.
```ts
import { createMigrationSequence, isShape } from 'tldraw'
const myMigrations = createMigrationSequence({
sequenceId: SEQUENCE_ID,
sequence: [
{
id: Versions.AddColor,
// Scope can be one of
// - 'store' to have the up function called on the whole snapshot at once
// - 'record' to have the up function called on each record individually
scope: 'record',
// if scope is 'record', you can filter which records the migration runs on
filter: (record) => isShape(record) && record.type === 'custom-shape',
up(record) {
record.props.color = 'black'
},
},
],
})
```
And finally pass your migrations in to tldraw via the `migrations` config option. There are a few places where you might need to do this, depending on how specialized your usage of Tldraw is:
```tsx
// When rendering the Tldraw component
<Tldraw
...
migrations={[myMigrations]}
/>
// or when creating the store
store = createTLStore({
...
migrations: [myMigrations],
})
// or when creating the schema
schema = createTLSchema({
...
migrations: [myMigrations],
})
```
### Updating legacy shape migrations (defineMigrations)
You can convert your legacy migrations to the new migrations format by the following process:
1. Wrap your version numbers in `createShapePropsMigrationIds`
```diff
- const Versions = {
+ const Versions = createShapePropMigrationIds('custom-shape', {
AddColor: 1
- }
+ })
```
2. Replace your `defineMigrations` call with `createShapePropsMigrationSequence`
```ts
const migrations = defineMigrations({
currentVersion: Versions.AddColor,
migrators: {
[Versions.AddColor]: {
up: (shape: any) => ({ ...shape, props: { ...shape.props, color: 'black' } }),
down: ({ props: { color, ...props }, ...shape }: any) => ({ ...shape, props }),
},
},
})
```
Becomes
```ts
const migrations = createShapePropsMigrationSequence({
sequence: [
{
id: Versions.AddColor,
// [!!!] You no longer have access to the top-level shape object.
// Only the shape.props object is passed in to the migrator function.
up(props) {
// [!!!] You no longer need to return a new copy of the shape object.
// Instead, you can modify the props object in place.
props.color = 'black'
},
// [!!!] You no longer need to specify a down migration.
},
],
})
```
## Examples
### Local persistence
Tldraw ships with a local-only sync engine based on `IndexedDb` and `BroadcastChannel` called [`TLLocalSyncClient`](https://github.com/tldraw/tldraw/blob/main/packages/editor/src/lib/utils/sync/TLLocalSyncClient.ts).
### Tldraw.com sync engine
[tldraw.com/r](https://tldraw.com/r) currently uses a simple custom sync engine based on a push/pull/rebase-style algorithm.
It can be found [here](https://github.com/tldraw/tldraw/tree/main/packages/tlsync).
It was optimized for Cloudflare workers with [DurableObjects](https://developers.cloudflare.com/durable-objects/)
We don't suggest using our code directly yet, but it may serve as a good reference for your own sync engine.
### Yjs sync example
We created a [tldraw-yjs example](https://github.com/tldraw/tldraw-yjs-example) to illustrate a way of using yjs with the tldraw SDK.
### Shape props migrations example
Our [custom-config example](/examples/shapes/tools/custom-config) shows how to add custom shape props migrations to the tldraw store.
### Meta properties migrations example
Our [custom-config example](/examples/shapes/tools/custom-config) shows how to add custom migrations to the tldraw store.

View file

@ -239,4 +239,6 @@ You can turn on `pointer-events` to allow users to interact inside of the shape.
You can make shapes "editable" to help decide when they're interactive or not.
...and more!
### Migrations
You can add migrations for your shape props by adding a `migrations` property to your shape's util class. See [the persistence docs](/docs/persistence#Shape-props-migrations) for more information.