this is take #2 of this PR https://github.com/tldraw/tldraw/pull/3745
As I look at LOD holistically and whether we have multiple sources when
working locally, I learned that our system used base64 encoding of
assets directly. Issue https://github.com/tldraw/tldraw/issues/3728
<img width="1350" alt="assetstore"
src="https://github.com/tldraw/tldraw/assets/469604/e7b41e29-6656-4d9b-b462-72d43b98f3f7">
The motivations and benefits are:
- store size: not having a huge base64 blobs injected in room data
- perf on loading snapshot: this helps with loading the room data more
quickly
- multiple sources: furthermore, if we do decide to have multiple
sources locally (for each asset), then we won't get a multiplicative
effect of even larger JSON blobs that have lots of base64 data in them
- encoding/decoding perf: this also saves the (slow) step of having to
base64 encode/decode our assets, we can just strictly with work with
blobs.
Todo:
- [x] decodes video and images
- [x] make sure it syncs to other tabs
- [x] make sure it syncs to other multiplayer room
- [x] fix tests
### Change Type
<!-- ❗ Please select a 'Scope' label ❗️ -->
- [x] `sdk` — Changes the tldraw SDK
- [ ] `dotcom` — Changes the tldraw.com web app
- [ ] `docs` — Changes to the documentation, examples, or templates.
- [ ] `vs code` — Changes to the vscode plugin
- [ ] `internal` — Does not affect user-facing stuff
<!-- ❗ Please select a 'Type' label ❗️ -->
- [ ] `bugfix` — Bug fix
- [ ] `feature` — New feature
- [x] `improvement` — Improving existing features
- [ ] `chore` — Updating dependencies, other boring stuff
- [ ] `galaxy brain` — Architectural changes
- [ ] `tests` — Changes to any test code
- [ ] `tools` — Changes to infrastructure, CI, internal scripts,
debugging tools, etc.
- [ ] `dunno` — I don't know
### Test Plan
1. Test the shit out of uploading/downloading video/image assets,
locally+multiplayer.
- [ ] Need to fix current tests and write new ones
### Release Notes
- Assets: store as reference to blob in indexedDB instead of storing
directly as base64 in the snapshot.
Previously, we had the `ae-forgotten-export` rule from api-extractor
disabled. This rule makes sure that everything that's referred to in the
public API is actually exported. There are more details on the rule
[here](https://api-extractor.com/pages/messages/ae-forgotten-export/),
but not exporting public API entires is bad because they're hard to
document and can't be typed/called from consumer code. For us, the big
effect is that they don't appear in our docs at all.
This diff re-enables that rule. Now, if you introduce something new to
the public API but don't export it, your build will fail.
### Change Type
- [x] `docs` — Changes to the documentation, examples, or templates.
- [x] `improvement` — Improving existing features
Typescript's type aliases (`type X = thing`) can refer to basically
anything, which makes it hard to write an automatic document formatter
for them. Interfaces on the other hand are only object, so they play
much nicer with docs. Currently, object-flavoured type aliases don't
really get expanded at all on our docs site, which means we have a bunch
of docs content that's not shown on the site.
This diff introduces a lint rule that forces `interface X {foo: bar}`s
instead of `type X = {foo: bar}` where possible, as it results in a much
better documentation experience:
Before:
<img width="437" alt="Screenshot 2024-05-22 at 15 24 13"
src="https://github.com/tldraw/tldraw/assets/1489520/32606fd1-6832-4a1e-aa5f-f0534d160c92">
After:
<img width="431" alt="Screenshot 2024-05-22 at 15 33 01"
src="https://github.com/tldraw/tldraw/assets/1489520/4e0d59ee-c38e-4056-b9fd-6a7f15d28f0f">
### Change Type
- [x] `sdk` — Changes the tldraw SDK
- [x] `docs` — Changes to the documentation, examples, or templates.
- [x] `improvement` — Improving existing features
Describe what your pull request does. If appropriate, add GIFs or images
showing the before and after.
### Change Type
<!-- ❗ Please select a 'Scope' label ❗️ -->
- [ ] `sdk` — Changes the tldraw SDK
- [ ] `dotcom` — Changes the tldraw.com web app
- [x] `docs` — Changes to the documentation, examples, or templates.
- [ ] `vs code` — Changes to the vscode plugin
- [ ] `internal` — Does not affect user-facing stuff
<!-- ❗ Please select a 'Type' label ❗️ -->
- [ ] `bugfix` — Bug fix
- [ ] `feature` — New feature
- [ ] `improvement` — Improving existing features
- [x] `chore` — Updating dependencies, other boring stuff
- [ ] `galaxy brain` — Architectural changes
- [ ] `tests` — Changes to any test code
- [ ] `tools` — Changes to infrastructure, CI, internal scripts,
debugging tools, etc.
- [ ] `dunno` — I don't know
### Test Plan
N/A
- [ ] Unit Tests
- [ ] End to end tests
### Release Notes
- Update example for Union type
---
I believe this type was changed and `literal` is what it should be now.
This PR updates readmes (including fixing some typos) and adds a link to
a Google Form for license inquiries.
### Change Type
- [x] `internal` — Does not affect user-facing stuff
- [x] `chore` — other boring stuff
These were needed when the docs lived in a different repo, but they
don't any more so we can get rid of them.
### Change Type
- [x] `internal` — Does not affect user-facing stuff
- [x] `chore` — Updating dependencies, other boring stuff
Describe what your pull request does. If appropriate, add GIFs or images
showing the before and after.
### Change Type
- [x] `sdk` — Changes the tldraw SDK
- [x] `galaxy brain` — Architectural changes
### Test Plan
1. Add a step-by-step description of how to test your PR here.
2.
- [ ] Unit Tests
- [ ] End to end tests
### Release Notes
#### BREAKING CHANGES
- The `Migrations` type is now called `LegacyMigrations`.
- The serialized schema format (e.g. returned by
`StoreSchema.serialize()` and `Store.getSnapshot()`) has changed. You
don't need to do anything about it unless you were reading data directly
from the schema for some reason. In which case it'd be best to avoid
that in the future! We have no plans to change the schema format again
(this time was traumatic enough) but you never know.
- `compareRecordVersions` and the `RecordVersion` type have both
disappeared. There is no replacement. These were public by mistake
anyway, so hopefully nobody had been using it.
- `compareSchemas` is a bit less useful now. Our migrations system has
become a little fuzzy to allow for simpler UX when adding/removing
custom extensions and 3rd party dependencies, and as a result we can no
longer compare serialized schemas in any rigorous manner. You can rely
on this function to return `0` if the schemas are the same. Otherwise it
will return `-1` if the schema on the right _seems_ to be newer than the
schema on the left, but it cannot guarantee that in situations where
migration sequences have been removed over time (e.g. if you remove one
of the builtin tldraw shapes).
Generally speaking, the best way to check schema compatibility now is to
call `store.schema.getMigrationsSince(persistedSchema)`. This will throw
an error if there is no upgrade path from the `persistedSchema` to the
current version.
- `defineMigrations` has been deprecated and will be removed in a future
release. For upgrade instructions see
https://tldraw.dev/docs/persistence#Updating-legacy-shape-migrations-defineMigrations
- `migrate` has been removed. Nobody should have been using this but if
you were you'll need to find an alternative. For migrating tldraw data,
you should stick to using `schema.migrateStoreSnapshot` and, if you are
building a nuanced sync engine that supports some amount of backwards
compatibility, also feel free to use `schema.migratePersistedRecord`.
- the `Migration` type has changed. If you need the old one for some
reason it has been renamed to `LegacyMigration`. It will be removed in a
future release.
- the `Migrations` type has been renamed to `LegacyMigrations` and will
be removed in a future release.
- the `SerializedSchema` type has been augmented. If you need the old
version specifically you can use `SerializedSchemaV1`
---------
Co-authored-by: Steve Ruiz <steveruizok@gmail.com>
## Migration path
1. If any of your shapes implement `toSvg` for exports, you'll need to
replace your implementation with a new version that returns JSX (it's a
react component) instead of manually constructing SVG DOM nodes
2. `editor.getSvg` is deprecated. It still works, but will be going away
in a future release. If you still need SVGs as DOM elements rather than
strings, use `new DOMParser().parseFromString(svgString,
'image/svg+xml').firstElementChild`
## The change in detail
At the moment, our SVG exports very carefully try to recreate the
visuals of our shapes by manually constructing SVG DOM nodes. On its own
this is really painful, but it also results in a lot of duplicated logic
between the `component` and `getSvg` methods of shape utils.
In #3020, we looked at using string concatenation & DOMParser to make
this a bit less painful. This works, but requires specifying namespaces
everywhere, is still pretty painful (no syntax highlighting or
formatting), and still results in all that duplicated logic.
I briefly experimented with creating my own version of the javascript
language that let you embed XML like syntax directly. I was going to
call it EXTREME JAVASCRIPT or XJS for short, but then I noticed that we
already wrote the whole of tldraw in this thing called react and a (imo
much worse named) version of the javascript xml thing already existed.
Given the entire library already depends on react, what would it look
like if we just used react directly for these exports? Turns out things
get a lot simpler! Take a look at lmk what you think
This diff was intended as a proof of concept, but is actually pretty
close to being landable. The main thing is that here, I've deliberately
leant into this being a big breaking change to see just how much code we
could delete (turns out: lots). We could if we wanted to make this
without making it a breaking change at all, but it would add back a lot
of complexity on our side and run a fair bit slower
---------
Co-authored-by: huppy-bot[bot] <128400622+huppy-bot[bot]@users.noreply.github.com>
Currently, we only use native `structuredClone` in the browser, falling
back to `JSON.parse(JSON.stringify(...))` elsewhere, despite Node
supporting `structuredClone` [since
v17](https://developer.mozilla.org/en-US/docs/Web/API/structuredClone)
and Cloudflare Workers supporting it [since
2022](https://blog.cloudflare.com/standards-compliant-workers-api/).
This PR adjusts our shim to use the native `structuredClone` on all
platforms, if available.
Additionally, `jsdom` doesn't implement `structuredClone`, a bug [open
since 2022](https://github.com/jsdom/jsdom/issues/3363). This PR patches
`jsdom` environment in all packages/apps that use it for tests.
Also includes a driveby removal of `deepCopy`, a function that is
strictly inferior to `structuredClone`.
### Change Type
<!-- ❗ Please select a 'Scope' label ❗️ -->
- [x] `sdk` — Changes the tldraw SDK
- [x] `dotcom` — Changes the tldraw.com web app
- [ ] `docs` — Changes to the documentation, examples, or templates.
- [ ] `vs code` — Changes to the vscode plugin
- [ ] `internal` — Does not affect user-facing stuff
<!-- ❗ Please select a 'Type' label ❗️ -->
- [ ] `bugfix` — Bug fix
- [ ] `feature` — New feature
- [x] `improvement` — Improving existing features
- [x] `chore` — Updating dependencies, other boring stuff
- [ ] `galaxy brain` — Architectural changes
- [ ] `tests` — Changes to any test code
- [ ] `tools` — Changes to infrastructure, CI, internal scripts,
debugging tools, etc.
- [ ] `dunno` — I don't know
### Test Plan
1. A smoke test would be enough
- [ ] Unit Tests
- [x] End to end tests
This PR bumps TypeScript to 5.3.3 and API extractor. We started getting
some weird behavior in CI due to different versions of the two
libraries, ie where the CI api.jsons would differ from those built
locally.
### Change Type
- [x] `dependencies` — Changes to package dependencies[^1]
Closes#2800
This PR makes it so that `check-scripts` will error out if you forget to
add a "references" entry to a tsconfig file when adding an internal
dependency in our monorepo.
If these project references are missed it can prevent TS from
building/rebuilding things when they need to be built/rebuilt.
### Change Type
- [x] `internal` — Any other changes that don't affect the published
package[^2]
Make sure we check if we have the optional function before calling it
### Change Type
- [x] `patch` — Bug fix
- [ ] `minor` — New feature
- [ ] `major` — Breaking change
- [ ] `dependencies` — Changes to package dependencies[^1]
- [ ] `documentation` — Changes to the documentation only[^2]
- [ ] `tests` — Changes to any test code only[^2]
- [ ] `internal` — Any other changes that don't affect the published
package[^2]
- [ ] I don't know
[^1]: publishes a `patch` release, for devDependencies use `internal`
[^2]: will not publish a new version
This PR adds a validation mode whereby previous known-to-be-valid values
can be used to speed up the validation process itself. At the same time
it enables us to do fine-grained equality checking on records much more
quickly than by using something like lodash isEqual, and using that we
can prevent triggering effects for record updates that don't actually
alter any values in the store.
Here's some preliminary perf testing of average time spent in
`store.put()` during some common interactions
| task | before (ms) | after (ms) |
| ---- | ---- | ---- |
| drawing lines | 0.0403 | 0.0214 |
| drawing boxes | 0.0408 | 0.0348 |
| translating lines | 0.0352 | 0.0042 |
| translating boxes | 0.0051 | 0.0032 |
| rotating lines | 0.0312 | 0.0065 |
| rotating boxes | 0.0053 | 0.0035 |
| brush selecting boxes | 0.0200 | 0.0232 |
| traversal with shapes | 0.0130 | 0.0108 |
| traversal without shapes | 0.0201 | 0.0173 |
**traversal** means moving the camera and pointer around the canvas
#### Discussion
At the scale of hundredths of a millisecond these .put operations are so
fast that even if they became literally instantaneous the change would
not be human perceptible. That said, there is an overall marked
improvement here. Especially for dealing with draw shapes.
These figures are also mostly in line with expectations, aside from a
couple of things:
- I don't understand why the `brush selecting boxes` task got slower
after the change.
- I don't understand why the `traversal` tasks are slower than the
`translating boxes` task, both before and after. I would expect that
.putting shape records would be much slower than .putting pointer/camera
records (since the latter have fewer and simpler properties)
### Change Type
- [x] `patch` — Bug fix
### Test Plan
1. Add a step-by-step description of how to test your PR here.
2.
- [ ] Unit Tests
- [ ] End to end tests
### Release Notes
- Add a brief release note for your PR here.
This PR refactors our menu systems and provides an interface to hide or
replace individual user interface elements.
# Background
Previously, we've had two types of overrides:
- "schema" overrides that would allow insertion or replacement of items
in the different menus
- "component" overrides that would replace components in the editor's
user interface
This PR is an attempt to unify the two and to provide for additional
cases where the "schema-based" user interface had begun to break down.
# Approach
This PR makes no attempt to change the `actions` or `tools`
overrides—the current system seems to be correct for those because they
are not reactive. The challenge with the other ui schemas is that they
_are_ reactive, and thus the overrides both need to a) be fed in from
outside of the editor as props, and b) react to changes from the editor,
which is an impossible situation.
The new approach is to use React to declare menu items. (Surprise!)
```tsx
function CustomHelpMenuContent() {
return (
<>
<DefaultHelpMenuContent />
<TldrawUiMenuGroup id="custom stuff">
<TldrawUiMenuItem
id="about"
label="Like my posts"
icon="external-link"
readonlyOk
onSelect={() => {
window.open('https://x.com/tldraw', '_blank')
}}
/>
</TldrawUiMenuGroup>
</>
)
}
const components: TLComponents = {
HelpMenuContent: CustomHelpMenuContent,
}
export default function CustomHelpMenuContentExample() {
return (
<div className="tldraw__editor">
<Tldraw components={components} />
</div>
)
}
```
We use a `components` prop with the combined editor and ui components.
- [ ] Create a "layout" component?
- [ ] Make UI components more isolated? If possible, they shouldn't
depend on styles outside of themselves, so that they can be used in
other layouts. Maybe we wait on this because I'm feeling a slippery
slope toward presumptions about configurability.
- [ ] OTOH maybe we go hard and consider these things as separate
components, even packages, with their own interfaces for customizability
/ configurability, just go all the way with it, and see what that looks
like.
# Pros
Top line: you can customize tldraw's user interface in a MUCH more
granular / powerful way than before.
It solves a case where menu items could not be made stateful from
outside of the editor context, and provides the option to do things in
the menus that we couldn't allow previously with the "schema-based"
approach.
It also may (who knows) be more performant because we can locate the
state inside of the components for individual buttons and groups,
instead of all at the top level above the "schema". Because items /
groups decide their own state, we don't have to have big checks on how
many items are selected, or whether we have a flippable state. Items and
groups themselves are allowed to re-build as part of the regular React
lifecycle. Menus aren't constantly being rebuilt, if that were ever an
issue.
Menu items can be shared between different menu types. We'll are
sometimes able to re-use items between, for example, the menu and the
context menu and the actions menu.
Our overrides no longer mutate anything, so there's less weird searching
and finding.
# Cons
This approach can make customization menu contents significantly more
complex, as an end user would need to re-declare most of a menu in order
to make any change to it. Luckily a user can add things to the top or
bottom of the context menu fairly easily. (And who knows, folks may
actually want to do deep customization, and this allows for it.)
It's more code. We are shipping more react components, basically one for
each menu item / group.
Currently this PR does not export the subcomponents, i.e. menu items. If
we do want to export these, then heaven help us, it's going to be a
_lot_ of exports.
# Progress
- [x] Context menu
- [x] Main menu
- [x] Zoom menu
- [x] Help menu
- [x] Actions menu
- [x] Keyboard shortcuts menu
- [x] Quick actions in main menu? (new)
- [x] Helper buttons? (new)
- [x] Debug Menu
And potentially
- [x] Toolbar
- [x] Style menu
- [ ] Share zone
- [x] Navigation zone
- [ ] Other zones
### Change Type
- [x] `major` — Breaking change
### Test Plan
1. use the context menu
2. use the custom context menu example
3. use cursor chat in the context menu
- [x] Unit Tests
- [ ] End to end tests
### Release Notes
- Add a brief release note for your PR here.
Currently, we type our fractional index keys as `string` and don't have
any validation for them. I'm touching some of this code for my work on
line handles and wanted to change that:
- fractional indexes are now `IndexKey`s, not `string`s. `IndexKey`s
have a brand property so can't be used interchangeably with strings
(like our IDs)
- There's a new `T.indexKey` validator which we can use in our
validations to make sure we don't end up with nonsense keys.
This PR is part of a series - please don't merge it until the things
before it have landed!
1. #2827 (you are here)
2. #2831
3. #2793
4. #2841
5. #2845
### Change Type
- [x] `patch` — Bug fix
### Test Plan
1. Mostly relying on unit & end to end tests here - no user facing
changes.
- [x] Unit Tests
As discussed offline, just making `yarn test` do what we expect it to.
### Change Type
- [x] `internal` — Any other changes that don't affect the published
package[^2]
This removes the ids from shape paths so that they can be grouped on our
error reporting tool.
### Change Type
- [x] `patch` — Bug fix
### Release Notes
- Error reporting: improve grouping for Sentry.
Biome as it is now didn't work out for us 😢
Summary for posterity:
* it IS much, much faster, fast enough to skip any sort of caching
* we couldn't fully replace Prettier just yet. We use Prettier
programmatically to format code in docs, and Biome's JS interface is
officially alpha and [had legacy peer deps
set](https://github.com/biomejs/biome/pull/1756) (which would fail our
CI build as we don't allow installation warnings)
* ternary formatting differs from Prettier, leading to a large diff
https://github.com/biomejs/biome/issues/1661
* import sorting differs from Prettier's
`prettier-plugin-organize-imports`, making the diff even bigger
* the deal breaker is a multi-second delay on saving large files (for us
it's
[Editor.ts](https://github.com/tldraw/tldraw/blob/main/packages/editor/src/lib/editor/Editor.ts))
in VSCode when import sorting is enabled. There is a seemingly relevant
Biome issue where I posted a small summary of our findings:
https://github.com/biomejs/biome/issues/1569#issuecomment-1930411623
Further actions:
* reevaluate in a few months as Biome matures
### Change Type
- [x] `internal` — Any other changes that don't affect the published
package
Biome seems to be MUCH faster than Prettier. Unfortunately, it
introduces some formatting changes around the ternary operator, so we
have to update files in the repo. To make revert easier if we need it,
the change is split into two PRs. This PR introduces a Biome CI check
and reformats all files accordingly.
## Change Type
- [x] `minor` — New feature
@si14 you might know a better way to wire this up! lemme know if there's
something more clever here.
### Change Type
- [x] `internal` — Any other changes that don't affect the published
package[^2]
### Release Notes
- Adds easier testing command for individual packages.
Allow urls for local files. This addresses the comment from
[here](https://github.com/tldraw/tldraw/pull/2428#issuecomment-1886221841).
### Change Type
- [x] `patch` — Bug fix
- [ ] `minor` — New feature
- [ ] `major` — Breaking change
- [ ] `dependencies` — Changes to package dependencies[^1]
- [ ] `documentation` — Changes to the documentation only[^2]
- [ ] `tests` — Changes to any test code only[^2]
- [ ] `internal` — Any other changes that don't affect the published
package[^2]
- [ ] I don't know
[^1]: publishes a `patch` release, for devDependencies use `internal`
[^2]: will not publish a new version
### Test Plan
1. Local images example should now work. We use images from the public
folder there.
This PR adds the docs app back into the tldraw monorepo.
## Deploying
We'll want to update our deploy script to update the SOURCE_SHA to the
newest release sha... and then deploy the docs pulling api.json files
from that release. We _could_ update the docs on every push to main, but
we don't have to unless something has changed. Right now there's no
automated deployments from this repo.
## Side effects
To make this one work, I needed to update the lock file. This might be
ok (new year new lock file), and everything builds as expected, though
we may want to spend some time with our scripts to be sure that things
are all good.
I also updated our prettier installation, which decided to add trailing
commas to every generic type. Which is, I suppose, [correct
behavior](https://github.com/prettier/prettier-vscode/issues/955)? But
that caused diffs in every file, which is unfortunate.
### Change Type
- [x] `internal` — Any other changes that don't affect the published
package[^2]
Adds validation for urls we use for our shapes and assets. This PR
includes a migration so we should check that existing rooms still load
correctly. There might be some that won't, but that means that they had
invalid url set.
### Change Type
- [x] `patch` — Bug fix
- [ ] `minor` — New feature
- [ ] `major` — Breaking change
- [ ] `dependencies` — Changes to package dependencies[^1]
- [ ] `documentation` — Changes to the documentation only[^2]
- [ ] `tests` — Changes to any test code only[^2]
- [ ] `internal` — Any other changes that don't affect the published
package[^2]
- [ ] I don't know
[^1]: publishes a `patch` release, for devDependencies use `internal`
[^2]: will not publish a new version
### Test Plan
1. Existing rooms should still load correctly (there should be no
validation errors).
2. Adding new images and videos should also work (test both local and
multiplayer rooms as they handle assets differently).
- [x] Unit Tests
- [ ] End to end tests
### Release Notes
- Add validation to urls.
---------
Co-authored-by: alex <alex@dytry.ch>
Changes `TRANDEMARK` -> `TRADEMARK` in docs links so they work.
### Change Type
- [ ] `patch` — Bug fix
- [ ] `minor` — New feature
- [ ] `major` — Breaking change
- [ ] `dependencies` — Changes to package dependencies[^1]
- [x] `documentation` — Changes to the documentation only[^2]
- [ ] `tests` — Changes to any test code only[^2]
- [ ] `internal` — Any other changes that don't affect the published
package[^2]
- [ ] I don't know
[^1]: publishes a `patch` release, for devDependencies use `internal`
[^2]: will not publish a new version
### Test Plan
- Confirm the links now work.
### Release Notes
- Fixes broken links in a number of docs files.
This PR updates the licenses across tldraw to a bespoke tldraw license.
The idea here is leverage dual licensing for revenue from companies
using tldraw. The source code and its distributions are provided under a
non-commercial license (tldraw) while we offer to sell / give out an
alternative exclusive-use license for companies who wish to use the
product for commercial purposes.
- [x] Add new license
- [x] Change licenses in package.jsons
- [x] Update READMEs
- [x] Update docs (separate repo PR)
- [x] Have alternative license in hand (US)
- [ ] Have alternative license in hand (UK)
- [x] Have sales contract in hand (US)
- [ ] Have sales contract in hand (UK)
### Change Type
- [x] `major` — Breaking change
This switches us from the 'alpha' to 'beta' prerelease tag. After we
make our first deploy folks will be able to install the latest version
by doing `npm install @tldraw/tldraw@beta`.
### Change Type
- [x] `internal` — Any other changes that don't affect the published
package[^2]
[^1]: publishes a `patch` release, for devDependencies use `internal`
[^2]: will not publish a new version
