Possible framework for a proof of concept

This is the fruits of about 3 attempts to write code that works. None of those attempts are here, but how edition 4 could work is at least documented now.

src/stores/room-list/README.md

@@ -0,0 +1,117 @@
+# Room list sorting
+
+It's so complicated it needs its own README.
+
+## Algorithms involved
+
+There's two main kinds of algorithms involved in the room list store: list ordering and tag sorting.
+Throughout the code an intentional decision has been made to call them the List Algorithm and Sorting
+Algorithm respectively. The list algorithm determines the behaviour of the room list whereas the sorting
+algorithm determines how individual tags (lists of rooms, sometimes called sublists) are ordered.
+
+Behaviour of the room list takes the shape of default sorting on tags in most cases, though it can
+override what is happening at the tag level depending on the algorithm used (as is the case with the
+importance algorithm, described later).
+
+Tag sorting is effectively the comparator supplied to the list algorithm. This gives the list algorithm
+the power to decide when and how to apply the tag sorting, if at all.
+
+### Tag sorting algorithm: Alphabetical
+
+When used, rooms in a given tag will be sorted alphabetically, where the alphabet is determined by a
+simple string comparison operation (essentially giving the browser the problem of figuring out if A
+comes before Z).
+
+### Tag sorting algorithm: Manual
+
+Manual sorting makes use of the `order` property present on all tags for a room, per the 
+[Matrix specification](https://matrix.org/docs/spec/client_server/r0.6.0#room-tagging). Smaller values
+of `order` cause rooms to appear closer to the top of the list.
+
+### Tag sorting algorithm: Recent
+
+Rooms are ordered by the timestamp of the most recent useful message. Usefulness is yet another algorithm
+in the room list system which determines whether an event type is capable of bubbling up in the room list.
+Normally events like room messages, stickers, and room security changes will be considered useful enough
+to cause a shift in time.
+
+Note that this is reliant on the event timestamps of the most recent message. Because Matrix is eventually
+consistent this means that from time to time a room might plummet or skyrocket across the tag due to the
+timestamp contained within the event (generated server-side by the sender's server).
+
+### List ordering algorithm: Natural
+
+This is the easiest of the algorithms to understand because it does essentially nothing. It imposes no
+behavioural changes over the tag sorting algorithm and is by far the simplest way to order a room list.
+Historically, it's been the only option in Riot and extremely common in most chat applications due to
+its relative deterministic behaviour.
+
+### List ordering algorithm: Importance
+
+On the other end of the spectrum, this is the most complicated algorithm which exists. There's major
+behavioural changes and the tag sorting algorithm is selectively applied depending on circumstances.
+
+Each tag which is not manually ordered gets split into 4 sections or "categories". Manually ordered tags
+simply get the manual sorting algorithm applied to them with no further involvement from the importance
+algorithm. There are 4 categories: Red, Grey, Bold, and Idle. Each has their own definition based off
+relative (perceived) importance to the user:
+
+* **Red**: The room has unread mentions waiting for the user.
+* **Grey**: The room has unread notifications waiting for the user. Notifications are simply unread
+  messages which cause a push notification or badge count. Typically this is the default as rooms are
+  set to 'All Messages'.
+* **Bold**: The room has unread messages waiting for the user. Essentially this is a grey room without
+  a badge/notification count (or 'Mentions Only'/'Muted').
+* **Idle**: No relevant activity has occurred in the room since the user last read it.
+
+Conveniently, each tag is ordered by those categories as presented: red rooms appear above grey, grey
+above idle, etc.
+
+Once the algorithm has determined which rooms belong in which categories, the tag sorting algorithm
+is applied to each category in a sub-sub-list fashion. This should result in the red rooms (for example)
+being sorted alphabetically amongst each other and the grey rooms sorted amongst each other, but 
+collectively the tag will be sorted into categories with red being at the top.
+
+The algorithm also has a concept of a 'sticky' room which is the room the user is currently viewing.
+The sticky room will remain in position on the room list regardless of other factors going on as typically
+clicking on a room will cause it to change categories into 'idle'. This is done by preserving N rooms
+above the selected room at all times where N is the number of rooms above the selected rooms when it was
+selected.
+
+For example, if the user has 3 red rooms and selects the middle room, they will always see exactly one
+room above their selection at all times. If they receive another notification and the tag ordering is set
+to Recent, they'll see the new notification go to the top position and the one that was previously there
+fall behind the sticky room.
+
+The sticky room's category is technically 'idle' while being viewed and is explicitly pulled out of the
+tag sorting algorithm's input as it must maintain its position in the list. When the user moves to another
+room, the previous sticky room is recalculated to determine which category it needs to be in as the user
+could have been scrolled up while new messages were received.
+
+Further, the sticky room is not aware of category boundaries and thus the user can see a shift in what 
+kinds of rooms move around their selection. An example would be the user having 4 red rooms, the user 
+selecting the third room (leaving 2 above it), and then having the rooms above it read on another device. 
+This would result in 1 red room and 1 other kind of room above the sticky room as it will try to maintain 
+2 rooms above the sticky room.
+
+An exception for the sticky room placement is when there's suddenly not enough rooms to maintain the placement
+exactly. This typically happens if the user selects a room and leaves enough rooms where it cannot maintain
+the N required rooms above the sticky room. In this case, the sticky room will simply decrease N as needed.
+The N value will never increase while selection remains unchanged: adding a bunch of rooms after having 
+put the sticky room in a position where it's had to decrease N will not increase N.
+
+## Responsibilities of the store
+
+The store is responsible for the ordering, upkeep, and tracking of all rooms. The component simply gets 
+an object containing the tags it needs to worry about and the rooms within. The room list component will 
+decide which tags need rendering (as it commonly filters out empty tags in most cases), and will deal with 
+all kinds of filtering.
+
+## Class breakdowns
+
+The `RoomListStore` is the major coordinator of various `IAlgorithm` implementations, which take care 
+of the various `ListAlgorithm` and `SortingAlgorithm` options. A `TagManager` is responsible for figuring 
+out which tags get which rooms, as Matrix specifies them as a reverse map: tags are defined on rooms and 
+are not defined as a collection of rooms (unlike how they are presented to the user). Various list-specific 
+utilities are also included, though they are expected to move somewhere more general when needed. For 
+example, the `membership` utilities could easily be moved elsewhere as needed.

src/stores/room-list/TagManager.ts

@@ -0,0 +1,32 @@
+/*
+Copyright 2020 The Matrix.org Foundation C.I.C.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+import {EventEmitter} from "events";
+
+// TODO: Docs on what this is
+export class TagManager extends EventEmitter {
+    constructor() {
+        super();
+    }
+
+    // TODO: Implementation.
+    // This will need to track where rooms belong in tags, and which tags they
+    // should be tracked within. This is algorithm independent because all the
+    // algorithms need to know what each tag contains.
+    //
+    // This will likely make use of the effective membership to determine the
+    // invite+historical sections.
+}

src/stores/room-list/algorithms/ChaoticAlgorithm.ts

@@ -31,6 +31,7 @@ export class ChaoticAlgorithm implements IAlgorithm {
     private rooms: Room[] = [];
 
     constructor(private representativeAlgorithm: ListAlgorithm) {
+        console.log("Constructed a ChaoticAlgorithm");
     }
 
     getOrderedRooms(): ITagMap {

src/stores/room-list/algorithms/IAlgorithm.ts

@@ -32,13 +32,6 @@ export enum ListAlgorithm {
     Natural = "NATURAL",
 }
 
-export enum Category {
-    Red = "RED",
-    Grey = "GREY",
-    Bold = "BOLD",
-    Idle = "IDLE",
-}
-
 export interface ITagSortingMap {
     // @ts-ignore - TypeScript really wants this to be [tagId: string] but we know better.
     [tagId: TagID]: SortAlgorithm;
@@ -50,7 +43,7 @@ export interface ITagMap {
 }
 
 // TODO: Convert IAlgorithm to an abstract class?
-// TODO: Add locking support to avoid concurrent writes
+// TODO: Add locking support to avoid concurrent writes?
 // TODO: EventEmitter support
 
 /**

src/stores/room-list/algorithms/ImportanceAlgorithm.ts

@@ -0,0 +1,189 @@
+/*
+Copyright 2020 The Matrix.org Foundation C.I.C.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+import { IAlgorithm, ITagMap, ITagSortingMap } from "./IAlgorithm";
+import { Room } from "matrix-js-sdk/src/models/room";
+import { isNullOrUndefined } from "matrix-js-sdk/src/utils";
+import { DefaultTagID, TagID } from "../models";
+import { splitRoomsByMembership } from "../membership";
+
+/**
+ * The determined category of a room.
+ */
+export enum Category {
+    /**
+     * The room has unread mentions within.
+     */
+    Red = "RED",
+    /**
+     * The room has unread notifications within. Note that these are not unread
+     * mentions - they are simply messages which the user has asked to cause a
+     * badge count update or push notification.
+     */
+    Grey = "GREY",
+    /**
+     * The room has unread messages within (grey without the badge).
+     */
+    Bold = "BOLD",
+    /**
+     * The room has no relevant unread messages within.
+     */
+    Idle = "IDLE",
+}
+
+/**
+ * An implementation of the "importance" algorithm for room list sorting. Where
+ * the tag sorting algorithm does not interfere, rooms will be ordered into
+ * categories of varying importance to the user. Alphabetical sorting does not
+ * interfere with this algorithm, however manual ordering does.
+ *
+ * The importance of a room is defined by the kind of notifications, if any, are
+ * present on the room. These are classified internally as Red, Grey, Bold, and
+ * Idle. Red rooms have mentions, grey have unread messages, bold is a less noisy
+ * version of grey, and idle means all activity has been seen by the user.
+ *
+ * The algorithm works by monitoring all room changes, including new messages in
+ * tracked rooms, to determine if it needs a new category or different placement
+ * within the same category. For more information, see the comments contained
+ * within the class.
+ */
+export class ImportanceAlgorithm implements IAlgorithm {
+
+    // HOW THIS WORKS
+    // --------------
+    //
+    // This block of comments assumes you've read the README one level higher.
+    // You should do that if you haven't already.
+    //
+    // Tags are fed into the algorithmic functions from the TagManager changes,
+    // which cause subsequent updates to the room list itself. Categories within
+    // those tags are tracked as index numbers within the array (zero = top), with
+    // each sticky room being tracked separately. Internally, the category index
+    // can be found from `this.indices[tag][category]` and the sticky room information
+    // from `this.stickyRooms[tag]`.
+    //
+    // Room categories are constantly re-evaluated and tracked in the `this.categorized`
+    // object. Note that this doesn't track rooms by category but instead by room ID.
+    // The theory is that by knowing the previous position, new desired position, and
+    // category indices we can avoid tracking multiple complicated maps in memory.
+    //
+    // The room list store is always provided with the `this.cached` results, which are
+    // updated as needed and not recalculated often. For example, when a room needs to
+    // move within a tag, the array in `this.cached` will be spliced instead of iterated.
+
+    private cached: ITagMap = {};
+    private sortAlgorithms: ITagSortingMap;
+    private rooms: Room[] = [];
+    private indices: {
+        // @ts-ignore - TS wants this to be a string but we know better than it
+        [tag: TagID]: {
+            // @ts-ignore - TS wants this to be a string but we know better than it
+            [category: Category]: number; // integer
+        };
+    } = {};
+    private stickyRooms: {
+        // @ts-ignore - TS wants this to be a string but we know better than it
+        [tag: TagID]: {
+            room?: Room;
+            nAbove: number; // integer
+        };
+    } = {};
+    private categorized: {
+        // @ts-ignore - TS wants this to be a string but we know better than it
+        [tag: TagID]: {
+            // TODO: Remove note
+            // Note: Should in theory be able to only track this by room ID as we'll know
+            // the indices of each category and can determine if a category needs changing
+            // in the cached list. Could potentially save a bunch of time if we can figure
+            // out where a room is supposed to be using offsets, some math, and leaving the
+            // array generally alone.
+            [roomId: string]: {
+                room: Room;
+                category: Category;
+            };
+        };
+    } = {};
+
+    constructor() {
+        console.log("Constructed an ImportanceAlgorithm");
+    }
+
+    getOrderedRooms(): ITagMap {
+        return this.cached;
+    }
+
+    async populateTags(tagSortingMap: ITagSortingMap): Promise<any> {
+        if (!tagSortingMap) throw new Error(`Map cannot be null or empty`);
+        this.sortAlgorithms = tagSortingMap;
+        this.setKnownRooms(this.rooms); // regenerate the room lists
+    }
+
+    handleRoomUpdate(room): Promise<boolean> {
+        return undefined;
+    }
+
+    setKnownRooms(rooms: Room[]): Promise<any> {
+        if (isNullOrUndefined(rooms)) throw new Error(`Array of rooms cannot be null`);
+        if (!this.sortAlgorithms) throw new Error(`Cannot set known rooms without a tag sorting map`);
+
+        this.rooms = rooms;
+
+        const newTags = {};
+        for (const tagId in this.sortAlgorithms) {
+            // noinspection JSUnfilteredForInLoop
+            newTags[tagId] = [];
+        }
+
+        // If we can avoid doing work, do so.
+        if (!rooms.length) {
+            this.cached = newTags;
+            return;
+        }
+
+        // TODO: Remove logging
+        const memberships = splitRoomsByMembership(rooms);
+        console.log({memberships});
+
+        // Step through each room and determine which tags it should be in.
+        // We don't care about ordering or sorting here - we're simply organizing things.
+        for (const room of rooms) {
+            const tags = room.tags;
+            let inTag = false;
+            for (const tagId in tags) {
+                // noinspection JSUnfilteredForInLoop
+                if (isNullOrUndefined(newTags[tagId])) {
+                    // skip the tag if we don't know about it
+                    continue;
+                }
+
+                inTag = true;
+
+                // noinspection JSUnfilteredForInLoop
+                newTags[tagId].push(room);
+            }
+
+            // If the room wasn't pushed to a tag, push it to the untagged tag.
+            if (!inTag) {
+                newTags[DefaultTagID.Untagged].push(room);
+            }
+        }
+
+        // TODO: Do sorting
+
+        // Finally, assign the tags to our cache
+        this.cached = newTags;
+    }
+}

src/stores/room-list/algorithms/index.ts

@@ -16,10 +16,11 @@ limitations under the License.
 
 import { IAlgorithm, ListAlgorithm } from "./IAlgorithm";
 import { ChaoticAlgorithm } from "./ChaoticAlgorithm";
+import { ImportanceAlgorithm } from "./ImportanceAlgorithm";
 
 const ALGORITHM_FACTORIES: { [algorithm in ListAlgorithm]: () => IAlgorithm } = {
     [ListAlgorithm.Natural]: () => new ChaoticAlgorithm(ListAlgorithm.Natural),
-    [ListAlgorithm.Importance]: () => new ChaoticAlgorithm(ListAlgorithm.Importance),
+    [ListAlgorithm.Importance]: () => new ImportanceAlgorithm(),
 };
 
 /**