PrivateCoffee/tldraw
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

Auto content refresh for docs site (#1606)

Browse source 
This PR improves the author experience when working on the docs site. 

When running `docs-dev`, the site's content will now update whenever a
content file is changed.

### Context

In the docs project, we generate content from two sources: from API
documentation generated by api-extractor and from markdown files in the
docs/content folder. Generating API docs is a relatively slow process
because it involves building and parsing TypeScript declaration files
for each package in the monorepo; however, generating docs from the
markdown files is basically instantaneous. The same script used to
address both tasks, which meant it was too slow to run on each save.
Instead, the script needed to be run manually or the dev server would
need to be restarted.

We now split the generation into two separate scripts. First, the script
runs to generate the API content; and then a second script runs to
generate the markdown content. The second script also imports and
combines the two sources of content. When we build the docs, both
scripts are run. When a markdown file changes, the new watcher only runs
the second script. This allows the site's content to be updated quickly
without having to generate the API docs each time.

Note that this does not incorporate live changes to package APIs, though
I can't think of a time where we be developing the docs and the APIs at
the same time.

### Change Type

- [x] `documentation` — Changes to the documentation only
This commit is contained in:
Steve Ruiz 2023-06-17 10:46:46 +01:00 committed by GitHub
parent 3f52c24fec
commit b9c6bf2fe8
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
11 changed files with 356 additions and 107 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
.eslintignore
View file

@ -20,6 +20,7 @@
apps/webdriver/www
apps/vscode/extension/editor
apps/examples/www
apps/docs/api-content.json
apps/docs/content.json
apps/vscode/extension/editor/index.js
apps/vscode/extension/editor/tldraw-assets.json

1
.prettierignore
View file

@ -9,6 +9,7 @@
**/.next/*
*.mdx
**/_archive/*
apps/docs/api-content.json
apps/docs/content.json
apps/webdriver/www/index.js
apps/vscode/extension/editor/*

1
apps/docs/.gitignore vendored
View file

@ -38,3 +38,4 @@ next-env.d.ts
content/gen
content.json
api-content.json

15
apps/docs/content/docs/introduction.mdx
View file

@ -12,17 +12,18 @@ tldraw is a React component that you can use to create infinite canvas experienc
![screenshot of tldraw](/images/screenshot.png)
These docs relate to tldraw's **alpha version**. This version is not yet open sourced, however it is available on npm and permissively licensed under Apache 2.0.
These docs relate to tldraw's **alpha version**, which is [open source](https://github.com/tldraw/tldraw), permissively licensed under Apache 2.0, and available [on npm](https://www.npmjs.com/package/@tldraw/tldraw) under the alpha and canary versions.
- Want to explore the code? Visit the [GitHub repo](https://github.com/tldraw/tldraw)).
- Want to dive in? Visit the [examples StackBlitz](https://stackblitz.com/github/tldraw/tldraw/tree/examples?file=src%2F1-basic%2FBasicExample.tsx).
- Found a bug or integration problem? Please create a ticket [here](https://github.com/tldraw/tldraw/issues).
- Questions or feedback? Let us know on the [Discord](https://discord.gg/JMbeb96jsh).
And if you are just looking for the regular app, try [tldraw.com](https://www.tldraw.com).
And if you are just looking for the regular tldraw app, try [tldraw.com](https://www.tldraw.com).
## Installation
First, install the `@tldraw/tldraw` package using `@alpha` for the **latest alpha release**. It also has peer dependencies on `signia` and `signia-react` which you will need to install at the same time.
First, install the `@tldraw/tldraw` package using `@alpha` for the **latest alpha release**. The package also has peer dependencies on `signia` and `signia-react` which you will need to install at the same time.
```bash
yarn add @tldraw/tldraw@alpha signia signia-react
@ -30,6 +31,14 @@ yarn add @tldraw/tldraw@alpha signia signia-react
npm install @tldraw/tldraw@alpha signia signia-react
```
To get the very latest version, use the [latest canary release](https://www.npmjs.com/package/@tldraw/tldraw?activeTab=versions). Docs for the very latest version are also available at [canary.tldraw.dev](https://canary.tldraw.dev).
```bash
yarn add @tldraw/tldraw@canary signia signia-react
# or
npm install @tldraw/tldraw@canary signia signia-react
```
## Usage
You should be able to use the `<Tldraw/>` component in any React editor.

2
apps/docs/package.json
View file

@ -27,7 +27,7 @@
"infinite"
],
"scripts": {
"dev": "next dev",
"dev": "yarn docs-content && NODE_ENV=development next-remote-watch ./content/**/*.mdx -c 'yarn run -T tsx ./scripts/generate-on-reload.ts'",
"build": "next build",
"start": "next start",
"lint": "yarn run -T tsx ../../scripts/lint.ts",

10
apps/docs/scripts/generate-on-reload.ts Normal file
View file

@ -0,0 +1,10 @@
// import { buildDocs } from './build-docs'
import { generateContent } from './generateContent'
async function main() {
const { log: nicelog } = console
nicelog('Creating content for www.')
await generateContent()
}
main()

128
apps/docs/scripts/generateApiContent.ts Normal file
View file

@ -0,0 +1,128 @@
import { ApiModel } from '@microsoft/api-extractor-model'
import fs from 'fs'
import path from 'path'
import { Articles, GeneratedContent, InputSection, MarkdownContent } from '../types/content-types'
import { generateSection } from './generateSection'
import { getApiMarkdown } from './getApiMarkdown'
import { getSlug } from './utils'
const { log: nicelog } = console
async function generateApiDocs() {
const apiInputSection: InputSection = {
id: 'gen' as string,
title: 'API',
description: "Reference for the tldraw package's APIs (generated).",
categories: [],
}
const addedCategories = new Set<string>()
const OUTPUT_DIR = path.join(process.cwd(), 'content', 'gen')
if (fs.existsSync(OUTPUT_DIR)) {
fs.rmdirSync(OUTPUT_DIR, { recursive: true })
}
fs.mkdirSync(OUTPUT_DIR)
// to include more packages in docs, add them to devDependencies in package.json
const packageJson = JSON.parse(fs.readFileSync(path.join(process.cwd(), 'package.json'), 'utf8'))
const tldrawPackagesToIncludeInDocs = Object.keys(packageJson.devDependencies).filter((dep) =>
dep.startsWith('@tldraw/')
)
const model = new ApiModel()
for (const packageName of tldrawPackagesToIncludeInDocs) {
// Get the file contents
const filePath = path.join(
process.cwd(),
'..',
'..',
'packages',
packageName.replace('@tldraw/', ''),
'api',
'api.json'
)
const packageModel = model.loadPackage(filePath)
const categoryName = packageModel.name.replace(`@tldraw/`, '')
if (!addedCategories.has(categoryName)) {
apiInputSection.categories!.push({
id: categoryName,
title: packageModel.name,
description: '',
groups: [
{
id: 'Namespace',
title: 'Namespaces',
},
{
id: 'Class',
title: 'Classes',
},
{
id: 'Function',
title: 'Functions',
},
{
id: 'Variable',
title: 'Variables',
},
{
id: 'Enum',
title: 'Enums',
},
{
id: 'Interface',
title: 'Interfaces',
},
{
id: 'TypeAlias',
title: 'TypeAliases',
},
],
})
addedCategories.add(categoryName)
}
const entrypoint = packageModel.entryPoints[0]
for (let j = 0; j < entrypoint.members.length; j++) {
const item = entrypoint.members[j]
const result = await getApiMarkdown(categoryName, item, j)
const outputFileName = `${getSlug(item)}.mdx`
fs.writeFileSync(path.join(OUTPUT_DIR, outputFileName), result.markdown)
}
}
return apiInputSection
}
export async function generateApiContent(): Promise<GeneratedContent> {
const content: MarkdownContent = {}
const articles: Articles = {}
try {
nicelog('• Generating api docs site content (content.json)')
const inputApiSection = await generateApiDocs()
const outputApiSection = generateSection(inputApiSection, content, articles)
const contentComplete = { sections: [outputApiSection], content, articles }
fs.writeFileSync(
path.join(process.cwd(), 'api-content.json'),
JSON.stringify(contentComplete, null, 2)
)
nicelog('✔ Generated api content.')
return contentComplete
} catch (error) {
nicelog(`x Could not generate site content.`)
throw error
}
}

112
apps/docs/scripts/generateContent.ts
View file

@ -1,4 +1,3 @@
import { ApiModel } from '@microsoft/api-extractor-model'
import fs from 'fs'
import matter from 'gray-matter'
import path from 'path'
@ -14,8 +13,6 @@ import {
Section,
Status,
} from '../types/content-types'
import { getApiMarkdown } from './getApiMarkdown'
import { getSlug } from './utils'
const { log: nicelog } = console
@ -191,109 +188,14 @@ function generateSection(
}
}
async function generateApiDocs() {
const apiInputSection: InputSection = {
id: 'gen' as string,
title: 'API',
description: "Reference for the tldraw package's APIs (generated).",
categories: [],
}
const addedCategories = new Set<string>()
const OUTPUT_DIR = path.join(process.cwd(), 'content', 'gen')
if (fs.existsSync(OUTPUT_DIR)) {
fs.rmdirSync(OUTPUT_DIR, { recursive: true })
}
fs.mkdirSync(OUTPUT_DIR)
// to include more packages in docs, add them to devDependencies in package.json
const packageJson = JSON.parse(fs.readFileSync(path.join(process.cwd(), 'package.json'), 'utf8'))
const tldrawPackagesToIncludeInDocs = Object.keys(packageJson.devDependencies).filter((dep) =>
dep.startsWith('@tldraw/')
)
const model = new ApiModel()
for (const packageName of tldrawPackagesToIncludeInDocs) {
// Get the file contents
const filePath = path.join(
process.cwd(),
'..',
'..',
'packages',
packageName.replace('@tldraw/', ''),
'api',
'api.json'
)
const packageModel = model.loadPackage(filePath)
const categoryName = packageModel.name.replace(`@tldraw/`, '')
if (!addedCategories.has(categoryName)) {
apiInputSection.categories!.push({
id: categoryName,
title: packageModel.name,
description: '',
groups: [
{
id: 'Namespace',
title: 'Namespaces',
},
{
id: 'Class',
title: 'Classes',
},
{
id: 'Function',
title: 'Functions',
},
{
id: 'Variable',
title: 'Variables',
},
{
id: 'Enum',
title: 'Enums',
},
{
id: 'Interface',
title: 'Interfaces',
},
{
id: 'TypeAlias',
title: 'TypeAliases',
},
],
})
addedCategories.add(categoryName)
}
const entrypoint = packageModel.entryPoints[0]
for (let j = 0; j < entrypoint.members.length; j++) {
const item = entrypoint.members[j]
const result = await getApiMarkdown(categoryName, item, j)
const outputFileName = `${getSlug(item)}.mdx`
fs.writeFileSync(path.join(OUTPUT_DIR, outputFileName), result.markdown)
}
}
return apiInputSection
}
export async function generateContent(): Promise<GeneratedContent> {
const content: MarkdownContent = {}
const articles: Articles = {}
const apiSection = await generateApiDocs()
nicelog('• Generating site content (content.json)')
try {
const outputSections: Section[] = [...(sections as InputSection[]), apiSection]
const outputSections: Section[] = [...(sections as InputSection[])]
.map((section) => generateSection(section, content, articles))
.filter((section) => section.categories.some((c) => c.articleIds.length > 0))
@ -301,7 +203,17 @@ export async function generateContent(): Promise<GeneratedContent> {
// Write to disk
const contentComplete = { sections: outputSections, content, articles }
const generatedApiContent = (await import(
path.join(process.cwd(), 'api-content.json')
)) as GeneratedContent
const contentComplete: GeneratedContent = {
sections: generatedApiContent
? [...outputSections, ...generatedApiContent.sections]
: outputSections,
content: generatedApiContent ? { ...content, ...generatedApiContent.content } : content,
articles: generatedApiContent ? { ...articles, ...generatedApiContent.articles } : articles,
}
fs.writeFileSync(
path.join(process.cwd(), 'content.json'),

171
apps/docs/scripts/generateSection.ts Normal file
View file

@ -0,0 +1,171 @@
import fs from 'fs'
import matter from 'gray-matter'
import path from 'path'
import authors from '../content/authors.json'
import {
Article,
Articles,
Category,
InputSection,
MarkdownContent,
Section,
Status,
} from '../types/content-types'
export function generateSection(
section: InputSection,
content: MarkdownContent,
articles: Articles
): Section {
// A temporary table of categories
const _categories: Record<string, Category> = {}
// Uncategorized articles
const _ucg: Article[] = []
// A temporary table of articles mapped to categories
const _categoryArticles: Record<string, Article[]> = Object.fromEntries(
section.categories.map((category) => [category.id, []])
)
// The file directory for this section
const dir = path.join(process.cwd(), 'content', section.id)
fs.readdirSync(dir, { withFileTypes: false }).forEach((result: string | Buffer) => {
try {
const filename = result.toString()
const fileContent = fs.readFileSync(path.join(dir, filename)).toString()
const extension = path.extname(filename)
const articleId = filename.replace(extension, '')
const parsed = matter({ content: fileContent }, {})
if (process.env.NODE_ENV !== 'development' && parsed.data.status !== 'published') {
return
}
// If a category was provided but that category was not found in the section, throw an error
const category =
parsed.data.category && section.categories.find((c) => c.id === parsed.data.category)
if (parsed.data.category && !category) {
throw Error(
`Could not find a category for section ${section.id} with id ${parsed.data.category}.`
)
}
if (parsed.data.author && !authors[parsed.data.author as keyof typeof authors]) {
throw Error(`Could not find an author with id ${parsed.data.author}.`)
}
// By default, the category is ucg (uncategorized)
const { category: categoryId = 'ucg' } = parsed.data
const article: Article = {
id: articleId,
sectionIndex: 0,
groupIndex: -1,
groupId: parsed.data.group ?? null,
categoryIndex: parsed.data.order ?? -1,
sectionId: section.id,
categoryId: parsed.data.category ?? 'ucg',
status: parsed.data.status ?? Status.Draft,
title: parsed.data.title ?? 'Article',
description: parsed.data.description ?? 'An article for the docs site.',
hero: parsed.data.hero ?? null,
date: parsed.data.date ? new Date(parsed.data.date).toISOString() : null,
keywords: parsed.data.keywords ?? [],
next: null,
prev: null,
author: parsed.data.author
? authors[parsed.data.author as keyof typeof authors] ?? null
: null,
sourceUrl: `https://github.com/tldraw/tldraw/tree/main/apps/docs/content/${section.id}/${articleId}${extension}`,
}
if (article.id === section.id) {
article.categoryIndex = -1
article.sectionIndex = -1
articles[section.id + '_index'] = article
content[section.id + '_index'] = parsed.content
} else {
if (category) {
if (article.id === category.id) {
article.categoryIndex = -1
article.sectionIndex = -1
articles[category.id + '_index'] = article
content[category.id + '_index'] = parsed.content
} else {
_categoryArticles[categoryId].push(article)
content[articleId] = parsed.content
}
} else {
_ucg.push(article)
content[articleId] = parsed.content
}
}
} catch (e) {
console.error(e)
}
})
const sortArticles = (articleA: Article, articleB: Article) => {
const { categoryIndex: categoryIndexA, date: dateA = '01/01/1970' } = articleA
const { categoryIndex: categoryIndexB, date: dateB = '01/01/1970' } = articleB
return categoryIndexA === categoryIndexB
? new Date(dateB!).getTime() > new Date(dateA!).getTime()
? 1
: -1
: categoryIndexA < categoryIndexB
? -1
: 1
}
let sectionIndex = 0
// Sort ucg articles by date and add them to the articles table
_ucg.sort(sortArticles).forEach((article, i) => {
article.categoryIndex = i
article.sectionIndex = sectionIndex++
article.prev = _ucg[i - 1]?.id ?? null
article.next = _ucg[i + 1]?.id ?? null
articles[article.id] = article
})
// Sort categorized articles by date and add them to the articles table
section.categories.forEach((category) => {
const categoryArticles = _categoryArticles[category.id]
categoryArticles.sort(sortArticles).forEach((article, i) => {
article.categoryIndex = i
article.sectionIndex = sectionIndex++
article.prev = categoryArticles[i - 1]?.id ?? null
article.next = categoryArticles[i + 1]?.id ?? null
articles[article.id] = article
})
_categories[category.id] = {
...category,
articleIds: categoryArticles.map((article) => article.id),
}
})
return {
...section,
categories: [
{
id: 'ucg',
title: 'Uncategorized',
description: 'Articles that do not belong to a category.',
groups: [],
articleIds: _ucg
.sort((a, b) => a.sectionIndex - b.sectionIndex)
.map((article) => article.id),
},
...section.categories.map(({ id }) => _categories[id]).filter((c) => c.articleIds.length > 0),
],
}
}

6
apps/docs/scripts/index.ts
View file

@ -1,10 +1,12 @@
// import { buildDocs } from './build-docs'
import { generateApiContent } from './generateApiContent'
import { generateContent } from './generateContent'
async function main() {
const { log } = console
log('Creating content for www.')
const { log: nicelog } = console
nicelog('Creating content for www.')
// await buildDocs()
await generateApiContent()
await generateContent()
}

14
apps/docs/types/content-types.ts
View file

@ -1,3 +1,17 @@
export type InputCategory = {
id: string
title: string
description: string
groups: Group[]
}
export type InputSection = {
id: string
title: string
description: string
categories: InputCategory[]
}
export enum Status {
Draft = 'draft',
Published = 'published',