Merge branch 'staging' into loading

CHANGELOG.md

@@ -1,20 +1,71 @@
-# X (3.23.0)
+# XerusDaamsi (3.23.0)
 
 ## Goals
 
+We plan to produce an updated installation guide for CryptPad instance administrators to coincide with the release of our 4.0.0 release. As we get closer to the end of the alphabet we're working to simplify the process of configuring instances. This release features several new admin panel features intended to supersede the usage of the server configuration file and provide the ability to modify instance settings at runtime.
+
+We also spent some time finalizing some major improvements to the history mode which is available in most of our document editors. More on that in the _Features_ section.
+
 ## Update notes
 
+This release introduces some behaviour which may require manual configuration on the part of the administrator. Read the following sections carefully or proceed at your own risk!
+
+### Automatic database maintenance
+
+When a user employs the _destroy_ functionality to make a pad unavailable it isn't typically deleted. Instead it is made unavailable by moving it into the server's archive directory. Archived files are intended to be removed after another configurable amount of time (`archiveRetentionTime` in your config file). The deletion of old files from your archive is handled by `evict-inactive.js`, which can be found in `cryptpad/scripts/`. Up until now this script needed to be run manually (typically as a cron job) with `node ./scripts/evict-inactive.js`. Since this isn't widely known we decided to integrate it directly into the server by automatically running the script once per day.
+
+The same _eviction_ process is also responsible for scanning your server's database for inactive documents (defined as those which haven't been accessed in a number of days specified in your config under `inactiveTime`). Such inactive documents are archived unless they have been stored within a registered users drive. Starting with this release we have added the ability to specify the number of days before an account will be considered inactive (`accountRetentionTime`). This will take into account whether they added any new documents to their drive, or whether any of the existing documents were accessed or modified by other users.
+
+If you prefer to run the eviction script manually you can disable its integration into the server by adding `disableIntegratedEviction: true` to your config file. An example is given in `cryptpad/config/config.example.js`. If you want this process to run manually you may set the same value to `false`, or comment it out if you prefer. Likewise, if you prefer to never remove accounts and their data due to account inactivity, you may also comment it out.
+
+If you haven't been manually running the eviction scripts we recommend that you carefully review all of the values mentioned above to ensure that you will not be surprised by the sudden and unintended removal of any data. As a reminder, they are:
+
+* `inactiveTime` (number of days before a file is considered inactive)
+* `archiveRetentionTime` (number of days that an archived file will be retained before it is permanently deleted)
+* `accountRetentionTime` (number of days of inactivity before an account is considered inactive and eligible for deletion)
+* `disableIntegratedEviction` (true if you prefer to run the eviction process manually or not at all, false or nothing if you want the server to handle eviction)
+
+### NGINX Configuration update
+
+After some testing on our part we've included an update to the example NGINX config file available in `cryptpad/docs/example.nginx.conf` which will enable a relatively new browser API which is required for XLSX export from our sheet editor. The relevant lines can be found beneath the comment `# Enable SharedArrayBuffer in Firefox (for .xlsx export)`.
+
+### Quota management
+
+Up until now the configuration file found in `cryptpad/config/config.js` has been the primary means of configuring a CryptPad instance. Unfortunately, as the server's behaviour becomes increasingly complex due to interest in a broad variety of use-cases this config file tends to grow. The kinds of questions that administrators ask via email, GitHub issues, and via our Matrix channel often suggest that admins haven't read through the comments in these files. Additionally, changes to the server's configuration can only be applied by restarting the server, which is increasingly disruptive as the service becomes more popular. To address these issues we've decided to start improving the instance admin panel such that it becomes the predominant means of modifying common server behaviours.
+
+We've started by making it possible to update storage settings from the _User storage_ section of the admin panel. Administrators can now update the default storage limit for users registered on the instance from the default quota of 50MB. It's also possible to allocate storage limits to particular users on the basis of their _Public Signing Key_, which can be found at the top of the _Accounts_ section on the settings page.
+
+Storage limits configured in this way will supercede those set via the server's config file, such that any modifications to a quota already set in the file will be ignored once you have modified or removed that user's quota via the admin panel. Admins are also able to view the parameters of all existing custom quotas loaded from either source.
+
+### How to update
+
+Once you've reviewed these settings and you're ready to update from 3.22.0 to 3.23.0:
+
+1. Modify your server's NGINX config file to include the new headers enabling XLSX export
+2. Stop CryptPad's nodejs server
+3. Get the latest platform code with git
+4. Install client-side dependencies with `bower update`
+5. Install server-side dependencies with `npm install`
+6. Reload NGINX with `service nginx reload` to apply its config changes
+7. Restart the CryptPad API server
+
 ## Features
 
-* responsive modals
-  * share
-  * access
-  * opencollective alert
-  * accessibility in alertify
-* remove inactive users
+
+* As mentioned in the update notes, this release features a server update which will enable XLSX export from our sheet editor in Firefox. XLSX files are generated entirely on the client, so all information will remain confidential, it only required a server update to enable a feature in Firefox which is required to perform the conversion.
+* We've also made some considerable improvements to the _history mode_ available in most of our document editors. We now display a more detailed timeline of changes according to who was present in the session, and group contiguous modifications made by a single user. Our intent is to provide an overview of the document's history which exposes the details which are most relevant to humans, rather than only allowing users to step through each individual change.
+* Another change which is related to our history mode improvements is support for "version links", which allow you to link to a specific historical version of a document while you scroll through the timeline of its modifications. You can also create _named snapshots_ of documents which will subsequently be displayed as highlights in the document's timeline.
+* Up until now we did not support _history mode_ for spreadsheets because our sheet integration is sufficiently different from our other editors that our existing history system could not be reused. That's still the case, but we've invested some time into creating a parallel history system with a slightly different user interface tailored to the display of sheet history.
+* Team owners and admins can now export team drives in the same manner as their own personal drives. The button to begin a full-drive export is available on the team's administration page.
+* During the summer we experimented with the idea of providing preview rendering options for more of the languages available in the code editor. We were particularly interested in providing LaTeX rendering in addition to Markdown. Unfortunately, it turned out to be a more complex feature than we have time for at the moment. In the process, however, we made it easier to integrate other rendering modes in addition to markdown. For the moment we've only added a simple rendering mode for displaying mixed HTML, but we'll consider using this framework to offer more options in the future.
+* While it might not be very noticeable depending on the size of the screen you use to view CryptPad we've spent some time making more of our interface responsive for mobile devices. You may notice this in particular on the modal menus used for sharing, setting access control parameters, and otherwise displaying alerts.
+* We've also begun improving support for screen-readers by adding the required HTML attributes to input fields and related markup. We'll continue to make incremental improvements regarding this and other accessibility issues that were raised during the third-party accessibility audit performed several months ago. This audit was performed on behalf of NLnet foundation (one of our major sponsors) as a part of their NGI Zero Privacy-Enhancing Technologies fund.
+* The _share modal_ from which users can generate shareable links already detects whether you have added any contacts on the platform and suggests how you can connect with them if you have not. We added this functionality some time late in 2019 since the same modal allowed users share documents directly with contacts and this mode became the subject of many support tickets. As it turns out, many users are now discovering _contact_ functionality via the _access modal_ through which you can add users to a document's allow list or delegate ownership. Since this has become a similar point of confusion we've added the same hints to make it a natural entry-point into CryptPad's social functionality.
 
 ## Bug fixes
 
+* We noticed that it was not possible for document owners to remove the extraneous history of old documents when those documents were protected by an _allow list_. This was due to the usage of an incorrect method for loading the document's metadata, leading to a false negative when testing if the user in question had sufficient access rights.
+* We also discovered an annoying bug in our filesystem storage APIs which caused the database adaptor to prevent scripts from terminating until several timeouts had finished running. These timeouts are now cancelled automatically so that the scripts stop running in a timely manner.
 
 # WoollyMammoth (3.22.0)
 

config/config.example.js

@@ -202,6 +202,15 @@ module.exports = {
      */
      //accountRetentionTime: 365,
 
+    /*  Starting with CryptPad 3.23.0, the server automatically runs
+     *  the script responsible for removing inactive data according to
+     *  your configured definition of inactivity. Set this value to `true`
+     *  if you prefer not to remove inactive data, or if you prefer to
+     *  do so manually using `scripts/evict-inactive.js`.
+     */
+    //disableIntegratedEviction: true,
+
+
     /*  Max Upload Size (bytes)
      *  this sets the maximum size of any one file uploaded to the server.
      *  anything larger than this size will be rejected

lib/api.js

@@ -1,13 +1,28 @@
 /* jshint esversion: 6 */
 const WebSocketServer = require('ws').Server;
 const NetfluxSrv = require('chainpad-server');
+const Decrees = require("./decrees");
 
-module.exports.create = function (config) {
+const nThen = require("nthen");
+
+module.exports.create = function (Env) {
+    var log = Env.Log;
+
+nThen(function (w) {
+    Decrees.load(Env, w(function (err) {
+        if (err) {
+            log.error('DECREES_LOADING', {
+                error: err.code || err,
+                message: err.message,
+            });
+            console.error(err);
+        }
+    }));
+}).nThen(function () {
     // asynchronously create a historyKeeper and RPC together
-    require('./historyKeeper.js').create(config, function (err, historyKeeper) {
+    require('./historyKeeper.js').create(Env, function (err, historyKeeper) {
         if (err) { throw err; }
 
-        var log = config.log;
 
         var noop = function () {};
 
@@ -21,7 +36,7 @@ module.exports.create = function (config) {
         };
 
         // spawn ws server and attach netflux event handlers
-        NetfluxSrv.create(new WebSocketServer({ server: config.httpServer}))
+        NetfluxSrv.create(new WebSocketServer({ server: Env.httpServer}))
             .on('channelClose', historyKeeper.channelClose)
             .on('channelMessage', historyKeeper.channelMessage)
             .on('channelOpen', historyKeeper.channelOpen)
@@ -50,4 +65,6 @@ module.exports.create = function (config) {
             })
             .register(historyKeeper.id, historyKeeper.directMessage);
     });
+});
+
 };

lib/commands/admin-rpc.js

@@ -207,20 +207,31 @@ the server adds two pieces of information to the supplied decree:
     }
 
     if (!changed) { return void cb(); }
+    Env.Log.info('ADMIN_DECREE', decree);
     Decrees.write(Env, decree, cb);
 };
 
 // CryptPad_AsyncStore.rpc.send('ADMIN', ['INSTANCE_STATUS], console.log)
 var instanceStatus = function (Env, Server, cb) {
     cb(void 0, {
-        restrictRegistration: Boolean(Env.restrictRegistration),
+        restrictRegistration: Env.restrictRegistration,
         launchTime: Env.launchTime,
         currentTime: +new Date(),
 
+        inactiveTime: Env.inactiveTime,
         accountRetentionTime: Env.accountRetentionTime,
         archiveRetentionTime: Env.archiveRetentionTime,
 
         defaultStorageLimit: Env.defaultStorageLimit,
+
+        lastEviction: Env.lastEviction,
+        // FIXME eviction is run in a worker and this isn't returned
+        //knownActiveAccounts: Env.knownActiveAccounts,
+        disableIntegratedEviction: Env.disableIntegratedEviction,
+        disableIntegratedTasks: Env.disableIntegratedTasks,
+
+        maxUploadSize: Env.maxUploadSize,
+        premiumUploadSize: Env.premiumUploadSize,
     });
 };
 

lib/commands/pin-rpc.js

@@ -114,6 +114,7 @@ Pinning.getTotalSize = function (Env, safeKey, cb) {
 /*  Users should be able to clear their own pin log with an authenticated RPC
 */
 Pinning.removePins = function (Env, safeKey, cb) {
+    // FIXME respect the queue
     Env.pinStore.removeChannel(safeKey, function (err) {
         Env.Log.info('DELETION_PIN_BY_OWNER_RPC', {
             safeKey: safeKey,

lib/commands/quota.js

@@ -28,7 +28,7 @@ Quota.isValidLimit = function (o) {
 Quota.applyCustomLimits = function (Env) {
     // DecreedLimits > customLimits > serverLimits;
 
-    // XXX perform an integrity check on shared limits
+    // FIXME perform an integrity check on shared limits
     // especially relevant because we use Env.limits
     // when considering whether to archive inactive accounts
 
@@ -117,13 +117,22 @@ Quota.queryAccountServer = function (Env, cb) {
     });
 };
 
-Quota.updateCachedLimits = function (Env, cb) {
-    Quota.applyCustomLimits(Env);
-    if (Env.blockDailyCheck === true ||
-        (typeof(Env.blockDailyCheck) === 'undefined' && Env.adminEmail === false && Env.allowSubscriptions === false)) {
-        return void cb();
-    }
+Quota.shouldContactServer = function (Env) {
+    return !(Env.blockDailyCheck === true ||
+        (
+            typeof(Env.blockDailyCheck) === 'undefined' &&
+            Env.adminEmail === false
+            && Env.allowSubscriptions === false
+        )
+    );
+};
 
+Quota.updateCachedLimits = function (Env, _cb) {
+    var cb = Util.mkAsync(_cb);
+
+    Quota.applyCustomLimits(Env);
+
+    if (!Quota.shouldContactServer(Env)) { return void cb(); }
     Quota.queryAccountServer(Env, function (err, json) {
         if (err) { return void cb(err); }
         if (!json) { return void cb(); }

lib/decrees.js

@@ -11,6 +11,19 @@ UPDATE_DEFAULT_STORAGE(<number>)
 SET_QUOTA(<string:signkey>, limit)
 RM_QUOTA(<string:signkey>)
 
+// INACTIVITY
+SET_INACTIVE_TIME
+SET_ACCOUNT_RETENTION_TIME
+SET_ARCHIVE_RETENTION_TIME
+
+// UPLOADS
+SET_MAX_UPLOAD_SIZE
+SET_PREMIUM_UPLOAD_SIZE
+
+// BACKGROUND PROCESSES
+DISABLE_INTEGRATED_TASKS
+DISABLE_INTEGRATED_EVICTION
+
 NOT IMPLEMENTED:
 
 // RESTRICTED REGISTRATION
@@ -19,14 +32,9 @@ REVOKE_INVITE
 REDEEM_INVITE
 
 // 2.0
-UPDATE_INACTIVE_TIME
-UPDATE_ACCOUNT_RETENTION_TIME
-UPDATE_ARCHIVE_RETENTION_TIME
-
-// 3.0
-UPDATE_MAX_UPLOAD_SIZE
-UPDATE_PREMIUM_UPLOAD_SIZE
-
+Env.adminEmail
+Env.supportMailbox
+Env.DEV_MODE || Env.FRESH_MODE,
 */
 
 var commands = {};
@@ -43,32 +51,78 @@ var commands = {};
 
 */
 
-// Toggles a simple boolean
-// CryptPad_AsyncStore.rpc.send('ADMIN', [ 'ADMIN_DECREE', ['RESTRICT_REGISTRATION', [true]]], console.log)
-commands.RESTRICT_REGISTRATION = function (Env, args) {
-    if (!Array.isArray(args) || typeof(args[0]) !== 'boolean') {
-        throw new Error('INVALID_ARGS');
-    }
-    var bool = args[0];
-    if (bool === Env.restrictRegistration) { return false; }
-    Env.restrictRegistration = bool;
-    return true;
+var args_isBoolean = function (args) {
+    return !(!Array.isArray(args) || typeof(args[0]) !== 'boolean');
 };
 
+// Toggles a simple boolean
+var makeBooleanSetter = function (attr) {
+    return function (Env, args) {
+        if (!args_isBoolean(args)) {
+            throw new Error('INVALID_ARGS');
+        }
+        var bool = args[0];
+        if (bool === Env[attr]) { return false; }
+        Env[attr] = bool;
+        return true;
+    };
+};
+
+// CryptPad_AsyncStore.rpc.send('ADMIN', [ 'ADMIN_DECREE', ['RESTRICT_REGISTRATION', [true]]], console.log)
+commands.RESTRICT_REGISTRATION = makeBooleanSetter('restrictRegistration');
+
+// CryptPad_AsyncStore.rpc.send('ADMIN', [ 'ADMIN_DECREE', ['DISABLE_INTEGRATED_EVICTION', [true]]], console.log)
+commands.DISABLE_INTEGRATED_EVICTION = makeBooleanSetter('disableIntegratedEviction');
+
+// CryptPad_AsyncStore.rpc.send('ADMIN', [ 'ADMIN_DECREE', ['DISABLE_INTEGRATED_TASKS', [true]]], console.log)
+commands.DISABLE_INTEGRATED_TASKS = makeBooleanSetter('disableIntegratedTasks');
+
+/*
 var isNonNegativeNumber = function (n) {
     return !(typeof(n) !== 'number' || isNaN(n) || n < 0);
 };
+*/
+
+var isInteger = function (n) {
+    return !(typeof(n) !== 'number' || isNaN(n) || (n % 1) !== 0);
+};
+
+var args_isInteger = function (args) {
+    return !(!Array.isArray(args) || !isInteger(args[0]));
+};
+
+var makeIntegerSetter = function (attr) {
+    return function (Env, args) {
+        if (!args_isInteger(args)) {
+            throw new Error('INVALID_ARGS');
+        }
+        var integer = args[0];
+        if (integer === Env[attr]) { return false; }
+        Env[attr] = integer;
+        return true;
+    };
+};
+
+// CryptPad_AsyncStore.rpc.send('ADMIN', [ 'ADMIN_DECREE', ['SET_MAX_UPLOAD_SIZE', [50 * 1024 * 1024]]], console.log)
+commands.SET_MAX_UPLOAD_SIZE = makeIntegerSetter('maxUploadSize');
+
+// CryptPad_AsyncStore.rpc.send('ADMIN', [ 'ADMIN_DECREE', ['SET_PREMIUM_UPLOAD_SIZE', [150 * 1024 * 1024]]], console.log)
+commands.SET_PREMIUM_UPLOAD_SIZE = makeIntegerSetter('premiumUploadSize');
 
 // CryptPad_AsyncStore.rpc.send('ADMIN', [ 'ADMIN_DECREE', ['UPDATE_DEFAULT_STORAGE', [100 * 1024 * 1024]]], console.log)
-commands.UPDATE_DEFAULT_STORAGE = function (Env, args) {
-    if (!Array.isArray(args) || !isNonNegativeNumber(args[0])) {
-        throw new Error('INVALID_ARGS');
-    }
-    var limit = args[0];
-    if (limit === Env.defaultStorageLimit) { return false; }
-    Env.defaultStorageLimit = limit;
-    return true;
-};
+commands.UPDATE_DEFAULT_STORAGE = makeIntegerSetter('defaultStorageLimit');
+
+// CryptPad_AsyncStore.rpc.send('ADMIN', [ 'ADMIN_DECREE', ['SET_LAST_EVICTION', [0]]], console.log)
+commands.SET_LAST_EVICTION = makeIntegerSetter('lastEviction');
+
+// CryptPad_AsyncStore.rpc.send('ADMIN', [ 'ADMIN_DECREE', ['SET_INACTIVE_TIME', [90]]], console.log)
+commands.SET_INACTIVE_TIME = makeIntegerSetter('inactiveTime');
+
+// CryptPad_AsyncStore.rpc.send('ADMIN', [ 'ADMIN_DECREE', ['SET_ARCHIVE_RETENTION_TIME', [30]]], console.log)
+commands.SET_ARCHIVE_RETENTION_TIME = makeIntegerSetter('archiveRetentionTime');
+
+// CryptPad_AsyncStore.rpc.send('ADMIN', [ 'ADMIN_DECREE', ['SET_ACCOUNT_RETENTION_TIME', [365]]], console.log)
+commands.SET_ACCOUNT_RETENTION_TIME = makeIntegerSetter('accountRetentionTime');
 
 var Quota = require("./commands/quota");
 var Keys = require("./keys");
@@ -173,9 +227,16 @@ var Schedule = require("./schedule");
 var Fse = require("fs-extra");
 var nThen = require("nthen");
 
-Decrees.load = function (Env, cb) {
+Decrees.load = function (Env, _cb) {
     Env.scheduleDecree = Env.scheduleDecree || Schedule();
 
+    var cb = Util.once(Util.mkAsync(function (err) {
+        if (err && err.code !== 'ENOENT') {
+            return void _cb(err);
+        }
+        _cb();
+    }));
+
     Env.scheduleDecree.blocking('', function (unblock) {
         var done = Util.once(Util.both(cb, unblock));
         nThen(function (w) {

lib/env.js

@@ -0,0 +1,201 @@
+/* jshint esversion: 6 */
+/* globals process */
+
+const Crypto = require('crypto');
+const WriteQueue = require("./write-queue");
+const BatchRead = require("./batch-read");
+
+const Keys = require("./keys");
+const Core = require("./commands/core");
+
+const Quota = require("./commands/quota");
+const Util = require("./common-util");
+
+module.exports.create = function (config) {
+
+    const Env = {
+        FRESH_KEY: '',
+        FRESH_MODE: true,
+        DEV_MODE: false,
+        configCache: {},
+        flushCache: function () {
+            Env.configCache = {};
+            Env.FRESH_KEY = +new Date();
+            if (!(Env.DEV_MODE || Env.FRESH_MODE)) { Env.FRESH_MODE = true; }
+            if (!Env.Log) { return; }
+            Env.Log.info("UPDATING_FRESH_KEY", Env.FRESH_KEY);
+        },
+
+        Log: undefined,
+        // store
+        id: Crypto.randomBytes(8).toString('hex'),
+
+        launchTime: +new Date(),
+
+        inactiveTime: config.inactiveTime,
+        archiveRetentionTime: config.archiveRetentionTime,
+        accountRetentionTime: config.accountRetentionTime,
+
+    // TODO implement mutability
+        adminEmail: config.adminEmail,
+        supportMailbox: config.supportMailboxPublicKey,
+
+        metadata_cache: {},
+        channel_cache: {},
+        queueStorage: WriteQueue(),
+        queueDeletes: WriteQueue(),
+        queueValidation: WriteQueue(),
+
+        batchIndexReads: BatchRead("HK_GET_INDEX"),
+        batchMetadata: BatchRead('GET_METADATA'),
+        batchRegisteredUsers: BatchRead("GET_REGISTERED_USERS"),
+        batchDiskUsage: BatchRead('GET_DISK_USAGE'),
+        batchUserPins: BatchRead('LOAD_USER_PINS'),
+        batchTotalSize: BatchRead('GET_TOTAL_SIZE'),
+        batchAccountQuery: BatchRead("QUERY_ACCOUNT_SERVER"),
+
+        intervals: {},
+        maxUploadSize: config.maxUploadSize || (20 * 1024 * 1024),
+        premiumUploadSize: false, // overridden below...
+        Sessions: {},
+        paths: {},
+        //msgStore: config.store,
+
+        netfluxUsers: {},
+
+        pinStore: undefined,
+
+        limits: {},
+        admins: [],
+        WARN: function (e, output) { // TODO deprecate this
+            if (!Env.Log) { return; }
+            if (e && output) {
+                Env.Log.warn(e, {
+                    output: output,
+                    message: String(e),
+                    stack: new Error(e).stack,
+                });
+            }
+        },
+
+        allowSubscriptions: config.allowSubscriptions === true,
+        blockDailyCheck: config.blockDailyCheck === true,
+
+        myDomain: config.myDomain,
+        mySubdomain: config.mySubdomain, // only exists for the accounts integration
+        customLimits: {},
+        // FIXME this attribute isn't in the default conf
+        // but it is referenced in Quota
+        domain: config.domain,
+
+        maxWorkers: config.maxWorkers,
+        disableIntegratedTasks: config.disableIntegratedTasks || false,
+        disableIntegratedEviction: config.disableIntegratedEviction || false,
+        lastEviction: +new Date(),
+        knownActiveAccounts: 0,
+    };
+
+    (function () {
+    // mode can be FRESH (default), DEV, or PACKAGE
+        if (process.env.PACKAGE) {
+        // `PACKAGE=1 node server` uses the version string from package.json as the cache string
+            //console.log("PACKAGE MODE ENABLED");
+            Env.FRESH_MODE = false;
+            Env.DEV_MODE = false;
+        } else if (process.env.DEV) {
+        // `DEV=1 node server` will use a random cache string on every page reload
+            //console.log("DEV MODE ENABLED");
+            Env.FRESH_MODE = false;
+            Env.DEV_MODE = true;
+        } else {
+        // `FRESH=1 node server` will set a random cache string when the server is launched
+        // and use it for the process lifetime or until it is reset from the admin panel
+            //console.log("FRESH MODE ENABLED");
+            Env.FRESH_KEY = +new Date();
+        }
+    }());
+
+
+
+
+    (function () {
+        var custom = config.customLimits;
+        if (!custom) { return; }
+
+        var stored = Env.customLimits;
+
+        Object.keys(custom).forEach(function (k) {
+            var unsafeKey = Keys.canonicalize(k);
+
+            if (!unsafeKey) {
+                console.log("INVALID_CUSTOM_LIMIT_ID", {
+                    message: "A custom quota upgrade was provided via your config with an invalid identifier. It will be ignored.",
+                    key: k,
+                    value: custom[k],
+                });
+                return;
+            }
+
+            if (stored[unsafeKey]) {
+                console.log("INVALID_CUSTOM_LIMIT_DUPLICATED", {
+                    message: "A duplicated custom quota upgrade was provided via your config which would have overridden an existing value. It will be ignored.",
+                    key: k,
+                    value: custom[k],
+                });
+                return;
+            }
+
+            if (!Quota.isValidLimit(custom[k])) {
+                console.log("INVALID_CUSTOM_LIMIT_VALUE", {
+                    message: "A custom quota upgrade was provided via your config with an invalid value. It will be ignored.",
+                    key: k,
+                    value: custom[k],
+                });
+                return;
+            }
+
+            var limit = stored[unsafeKey] = Util.clone(custom[k]);
+            limit.origin = 'config';
+        });
+    }());
+
+    (function () {
+        var pes = config.premiumUploadSize;
+        if (!isNaN(pes) && pes >= Env.maxUploadSize) {
+            Env.premiumUploadSize = pes;
+        }
+    }());
+
+    var paths = Env.paths;
+
+    var keyOrDefaultString = function (key, def) {
+        return typeof(config[key]) === 'string'? config[key]: def;
+    };
+
+    paths.pin = keyOrDefaultString('pinPath', './pins');
+    paths.block = keyOrDefaultString('blockPath', './block');
+    paths.data = keyOrDefaultString('filePath', './datastore');
+    paths.staging = keyOrDefaultString('blobStagingPath', './blobstage');
+    paths.blob = keyOrDefaultString('blobPath', './blob');
+    paths.decree = keyOrDefaultString('decreePath', './data/');
+    paths.archive = keyOrDefaultString('archivePath', './data/archive');
+    paths.task = keyOrDefaultString('taskPath', './tasks');
+
+    Env.defaultStorageLimit = typeof(config.defaultStorageLimit) === 'number' && config.defaultStorageLimit >= 0?
+        config.defaultStorageLimit:
+        Core.DEFAULT_LIMIT;
+
+    try {
+        Env.admins = (config.adminKeys || []).map(function (k) {
+            try {
+                return Keys.canonicalize(k);
+            } catch (err) {
+                return;
+            }
+        }).filter(Boolean);
+    } catch (e) {
+        console.error("Can't parse admin keys. Please update or fix your config.js file!");
+    }
+
+    return Env;
+};

lib/eviction.js

@@ -2,6 +2,7 @@ var nThen = require("nthen");
 var Bloom = require("@mcrowe/minibloom");
 var Util = require("../lib/common-util");
 var Pins = require("../lib/pins");
+var Keys = require("./keys");
 
 var getNewestTime = function (stats) {
     return stats[['atime', 'ctime', 'mtime'].reduce(function (a, b) {
@@ -42,12 +43,11 @@ module.exports = function (Env, cb) {
     // pre-converted to the 'safeKey' format so we can easily compare
     // them against ids we see on the filesystem
     var premiumSafeKeys = Object.keys(Env.limits || {})
-        .filter(function (key) {
-            return key.length === 44;
+        .map(function (id) {
+            return Keys.canonicalize(id);
         })
-        .map(function (unsafeKey) {
-            return Util.escapeKeyCharacters(unsafeKey);
-        });
+        .filter(Boolean)
+        .map(Util.escapeKeyCharacters);
 
     // files which have not been changed since before this date can be considered inactive
     var inactiveTime = +new Date() - (Env.inactiveTime * 24 * 3600 * 1000);
@@ -291,7 +291,7 @@ module.exports = function (Env, cb) {
             return activeDocs.test(docId);
         };
 
-        var accountIsActive = function (mtime, pinList, id) {
+        var accountIsActive = function (mtime, pinList) {
             // console.log("id [%s] in premiumSafeKeys", id, premiumSafeKeys.indexOf(id) !== -1);
             // if their pin log has changed recently then consider them active
             if (mtime && mtime > accountRetentionTime) {
@@ -299,11 +299,10 @@ module.exports = function (Env, cb) {
             }
 
             // iterate over their pinned documents until you find one that has been active
-            if (pinList.some(docIsActive)) {
-                return true;
-            }
+            return pinList.some(docIsActive);
+        };
 
-            // Finally, make sure it's not a premium account
+        var isPremiumAccount = function (id) {
             return premiumSafeKeys.indexOf(id) !== -1;
         };
 
@@ -317,7 +316,7 @@ module.exports = function (Env, cb) {
             var mtime = content.latest;
             var pinList = Object.keys(content.pins);
 
-            if (accountIsActive(mtime, pinList, id)) {
+            if (accountIsActive(mtime, pinList)) {
             // add active accounts' pinned documents to a second bloom filter
                 pinAll(pinList);
                 return void next();
@@ -332,6 +331,15 @@ module.exports = function (Env, cb) {
                 return void next();
             }
 
+            if (isPremiumAccount(id)) {
+                Log.info("EVICT_INACTIVE_PREMIUM_ACCOUNT", {
+                    id: id,
+                    mtime: mtime,
+                });
+                pinAll(pinList);
+                return void next();
+            }
+
             // remove the pin logs of inactive accounts if inactive account removal is configured
             pinStore.archiveChannel(id, function (err) {
                 if (err) {
@@ -348,6 +356,9 @@ module.exports = function (Env, cb) {
                 "EVICT_COUNT_ACCOUNTS":
                 "EVICT_INACTIVE_ACCOUNTS";
 
+            // update the number of known active accounts in Env for statistics
+            Env.knownActiveAccounts = accounts - inactive;
+
             Log.info(label, {
                 accounts: accounts,
                 inactive: inactive,

lib/historyKeeper.js

@@ -1,168 +1,18 @@
 /* jshint esversion: 6 */
 
 const nThen = require('nthen');
-const Crypto = require('crypto');
-const WriteQueue = require("./write-queue");
-const BatchRead = require("./batch-read");
 const RPC = require("./rpc");
 const HK = require("./hk-util.js");
-const Core = require("./commands/core");
-
-const Keys = require("./keys");
-const Quota = require("./commands/quota");
-const Util = require("./common-util");
-
 const Store = require("./storage/file");
 const BlobStore = require("./storage/blob");
 const Workers = require("./workers/index");
+const Core = require("./commands/core");
 
-module.exports.create = function (config, cb) {
-    const Log = config.log;
-    var WARN = function (e, output) {
-        if (e && output) {
-            Log.warn(e, {
-                output: output,
-                message: String(e),
-                stack: new Error(e).stack,
-            });
-        }
-    };
-
+module.exports.create = function (Env, cb) {
+    const Log = Env.Log;
     Log.silly('HK_LOADING', 'LOADING HISTORY_KEEPER MODULE');
 
-    const Env = {
-        Log: Log,
-        // store
-        id: Crypto.randomBytes(8).toString('hex'),
-
-        launchTime: +new Date(),
-
-        inactiveTime: config.inactiveTime,
-        archiveRetentionTime: config.archiveRetentionTime,
-        accountRetentionTime: config.accountRetentionTime,
-
-        metadata_cache: {},
-        channel_cache: {},
-        queueStorage: WriteQueue(),
-        queueDeletes: WriteQueue(),
-        queueValidation: WriteQueue(),
-
-        batchIndexReads: BatchRead("HK_GET_INDEX"),
-        batchMetadata: BatchRead('GET_METADATA'),
-        batchRegisteredUsers: BatchRead("GET_REGISTERED_USERS"),
-        batchDiskUsage: BatchRead('GET_DISK_USAGE'),
-        batchUserPins: BatchRead('LOAD_USER_PINS'),
-        batchTotalSize: BatchRead('GET_TOTAL_SIZE'),
-        batchAccountQuery: BatchRead("QUERY_ACCOUNT_SERVER"),
-
-        //historyKeeper: config.historyKeeper,
-        intervals: config.intervals || {},
-        maxUploadSize: config.maxUploadSize || (20 * 1024 * 1024),
-        premiumUploadSize: false, // overridden below...
-        Sessions: {},
-        paths: {},
-        //msgStore: config.store,
-
-        netfluxUsers: {},
-
-        pinStore: undefined,
-
-        limits: {},
-        admins: [],
-        WARN: WARN,
-        flushCache: config.flushCache,
-        adminEmail: config.adminEmail,
-        allowSubscriptions: config.allowSubscriptions === true,
-        blockDailyCheck: config.blockDailyCheck === true,
-
-        myDomain: config.myDomain,
-        mySubdomain: config.mySubdomain, // only exists for the accounts integration
-        customLimits: {},
-        // FIXME this attribute isn't in the default conf
-        // but it is referenced in Quota
-        domain: config.domain
-    };
-
-    (function () {
-        var custom = config.customLimits;
-        if (!custom) { return; }
-
-        var stored = Env.customLimits;
-
-        Object.keys(custom).forEach(function (k) {
-            var unsafeKey = Keys.canonicalize(k);
-
-            if (!unsafeKey) {
-                Log.warn("INVALID_CUSTOM_LIMIT_ID", {
-                    message: "A custom quota upgrade was provided via your config with an invalid identifier. It will be ignored.",
-                    key: k,
-                    value: custom[k],
-                });
-                return;
-            }
-
-            if (stored[unsafeKey]) {
-                Log.warn("INVALID_CUSTOM_LIMIT_DUPLICATED", {
-                    message: "A duplicated custom quota upgrade was provided via your config which would have overridden an existing value. It will be ignored.",
-                    key: k,
-                    value: custom[k],
-                });
-                return;
-            }
-
-            if (!Quota.isValidLimit(custom[k])) {
-                Log.warn("INVALID_CUSTOM_LIMIT_VALUE", {
-                    message: "A custom quota upgrade was provided via your config with an invalid value. It will be ignored.",
-                    key: k,
-                    value: custom[k],
-                });
-                return;
-            }
-
-            var limit = stored[unsafeKey] = Util.clone(custom[k]);
-            limit.origin = 'config';
-        });
-    }());
-
-    (function () {
-        var pes = config.premiumUploadSize;
-        if (!isNaN(pes) && pes >= Env.maxUploadSize) {
-            Env.premiumUploadSize = pes;
-        }
-    }());
-
-    var paths = Env.paths;
-
-    var keyOrDefaultString = function (key, def) {
-        return typeof(config[key]) === 'string'? config[key]: def;
-    };
-
-    var pinPath = paths.pin = keyOrDefaultString('pinPath', './pins');
-    paths.block = keyOrDefaultString('blockPath', './block');
-    paths.data = keyOrDefaultString('filePath', './datastore');
-    paths.staging = keyOrDefaultString('blobStagingPath', './blobstage');
-    paths.blob = keyOrDefaultString('blobPath', './blob');
-    paths.decree = keyOrDefaultString('decreePath', './data/');
-
-    Env.defaultStorageLimit = typeof(config.defaultStorageLimit) === 'number' && config.defaultStorageLimit >= 0?
-        config.defaultStorageLimit:
-        Core.DEFAULT_LIMIT;
-
-    try {
-        // XXX this should be the same as is exposed in server.js
-        // /api/config.adminKeys
-        Env.admins = (config.adminKeys || []).map(function (k) {
-            try {
-                return Keys.canonicalize(k);
-            } catch (err) {
-                return;
-            }
-        }).filter(Boolean);
-    } catch (e) {
-        console.error("Can't parse admin keys. Please update or fix your config.js file!");
-    }
-
-    config.historyKeeper = Env.historyKeeper = {
+    Env.historyKeeper = {
         metadata_cache: Env.metadata_cache,
         channel_cache: Env.channel_cache,
 
@@ -262,6 +112,8 @@ module.exports.create = function (config, cb) {
 
     Log.verbose('HK_ID', 'History keeper ID: ' + Env.id);
 
+    var pinPath = Env.paths.pin;
+
     nThen(function (w) {
         // create a pin store
         Store.create({
@@ -272,18 +124,20 @@ module.exports.create = function (config, cb) {
         }));
 
         // create a channel store
-        Store.create(config, w(function (err, _store) {
+        Store.create({
+            filePath: Env.paths.data,
+            archivepath: Env.paths.archive,
+        }, w(function (err, _store) {
             if (err) { throw err; }
-            config.store = _store;
             Env.msgStore = _store; // API used by rpc
             Env.store = _store; // API used by historyKeeper
         }));
 
         // create a blob store
         BlobStore.create({
-            blobPath: config.blobPath,
-            blobStagingPath: config.blobStagingPath,
-            archivePath: config.archivePath,
+            blobPath: Env.paths.blob,
+            blobStagingPath: Env.paths.staging,
+            archivePath: Env.paths.archive,
             getSession: function (safeKey) {
                 return Core.getSession(Env.Sessions, safeKey);
             },
@@ -293,32 +147,27 @@ module.exports.create = function (config, cb) {
         }));
     }).nThen(function (w) {
         Workers.initialize(Env, {
-            blobPath: config.blobPath,
-            blobStagingPath: config.blobStagingPath,
-            taskPath: config.taskPath,
-            pinPath: pinPath,
-            filePath: config.filePath,
-            archivePath: config.archivePath,
-            channelExpirationMs: config.channelExpirationMs,
-            verbose: config.verbose,
-            openFileLimit: config.openFileLimit,
+            blobPath: Env.paths.blob,
+            blobStagingPath: Env.paths.staging,
+            taskPath: Env.paths.task,
+            pinPath: Env.paths.pin,
+            filePath: Env.paths.data,
+            archivePath: Env.paths.archive,
 
-            inactiveTime: config.inactiveTime,
-            archiveRetentionTime: config.archiveRetentionTime,
-            accountRetentionTime: config.accountRetentionTime,
+            inactiveTime: Env.inactiveTime,
+            archiveRetentionTime: Env.archiveRetentionTime,
+            accountRetentionTime: Env.accountRetentionTime,
 
-            maxWorkers: config.maxWorkers,
+            maxWorkers: Env.maxWorkers,
         }, w(function (err) {
             if (err) {
                 throw new Error(err);
             }
         }));
     }).nThen(function () {
-        config.intervals = config.intervals || {};
-        if (config.disableIntegratedTasks) { return; }
-
         var tasks_running;
-        config.intervals.taskExpiration = setInterval(function () {
+        Env.intervals.taskExpiration = setInterval(function () {
+            if (Env.disableIntegratedTasks) { return; }
             if (tasks_running) { return; }
             tasks_running = true;
             Env.runTasks(function (err) {
@@ -329,19 +178,18 @@ module.exports.create = function (config, cb) {
             });
         }, 1000 * 60 * 5); // run every five minutes
     }).nThen(function () {
-        if (config.disableIntegratedEviction) { return; }
         const ONE_DAY = 24 * 1000 * 60 * 60;
         // setting the time of the last eviction to "now"
         // effectively makes it so that we'll start evicting after the server
         // has been up for at least one day
-        var last_eviction = +new Date();
 
         var active = false;
-        config.intervals.eviction = setInterval(function () {
+        Env.intervals.eviction = setInterval(function () {
+            if (Env.disableIntegratedEviction) { return; }
             if (active) { return; }
             var now = +new Date();
             // evict inactive data once per day
-            if (last_eviction && (now - ONE_DAY) < last_eviction) { return; }
+            if ((now - ONE_DAY) < Env.lastEviction) { return; }
             active = true;
             Env.evictInactive(function (err) {
                 if (err) {
@@ -349,28 +197,16 @@ module.exports.create = function (config, cb) {
                     Log.error('EVICT_INACTIVE_MAIN_ERROR', err);
                 }
                 active = false;
-                last_eviction = now;
+                Env.lastEviction = now;
             });
         }, 60 * 1000);
     }).nThen(function () {
-        var Decrees = require("./decrees");
-
-        Decrees.load(Env, function (err) {
-            if (err && err.code !== "ENOENT") {
-                Log.error('DECREES_LOADING', {
-                    error: err.code || err,
-                    message: err.message,
-                });
-                console.error(err);
-            }
-        });
-    }).nThen(function () {
 
         RPC.create(Env, function (err, _rpc) {
             if (err) { throw err; }
 
             Env.rpc = _rpc;
-            cb(void 0, config.historyKeeper);
+            cb(void 0, Env.historyKeeper);
         });
     });
 };

lib/storage/blob.js

@@ -204,14 +204,12 @@ var tryId = function (path, cb) {
     Fs.access(path, Fs.constants.R_OK | Fs.constants.W_OK, function (e) {
         if (!e) {
             // generate a new id (with the same prefix) and recurse
-            //WARN('ownedUploadComplete', 'id is already used '+ id);
             return void cb('EEXISTS');
         } else if (e.code === 'ENOENT') {
             // no entry, so it's safe for us to proceed
             return void cb();
         } else {
             // it failed in an unexpected way. log it
-            //WARN('ownedUploadComplete', e);
             return void cb(e.code);
         }
     });
@@ -229,7 +227,6 @@ var owned_upload_complete = function (Env, safeKey, id, cb) {
     }
 
     if (!isValidId(id)) {
-        //WARN('ownedUploadComplete', "id is invalid");
         return void cb('EINVAL_ID');
     }
 

lib/storage/file.js

@@ -779,9 +779,11 @@ const messageBin = (env, chanName, msgBin, cb) => {
         chan.writeStream.write(msgBin, function () {
             chan.onError.splice(chan.onError.indexOf(complete), 1);
             complete();
+// It seems like this reintroduces a file descriptor leak
             if (chan.onError.length) { return; }
             if (chan.delayClose && chan.delayClose.clear) {
                 chan.delayClose.clear();
+                destroyStream(chan.writeStream, chanName);
                 delete env.channels[chanName];
             }
         });
@@ -996,8 +998,6 @@ module.exports.create = function (conf, _cb) {
         root: conf.filePath || './datastore',
         archiveRoot: conf.archivePath || './data/archive',
         channels: { },
-        channelExpirationMs: conf.channelExpirationMs || 30000,
-        verbose: conf.verbose,
         batchGetChannel: BatchRead('store_batch_channel'),
     };
     var it;

scripts/evict-inactive.js

@@ -4,20 +4,41 @@ var Store = require("../lib/storage/file");
 var BlobStore = require("../lib/storage/blob");
 
 var Quota = require("../lib/commands/quota");
+var Environment = require("../lib/env");
+var Decrees = require("../lib/decrees");
 
 var config = require("../lib/load-config");
-var Env = {
-    inactiveTime: config.inactiveTime,
-    archiveRetentionTime: config.archiveRetentionTime,
-    accountRetentionTime: config.accountRetentionTime,
-    paths: {
-        pin: config.pinPath,
-    },
+
+var Env = Environment.create(config);
+
+var loadPremiumAccounts = function (Env, cb) {
+    nThen(function (w) {
+        // load premium accounts
+        Quota.updateCachedLimits(Env, w(function (err) {
+            if (err) {
+                Env.Log.error('EVICT_LOAD_PREMIUM_ACCOUNTS', {
+                    error: err,
+                });
+            }
+        }));
+    }).nThen(function (w) {
+        // load and apply decrees
+        Decrees.load(Env, w(function (err) {
+            if (err) {
+                Env.Log.error('EVICT_LOAD_DECREES', {
+                    error: err.code || err,
+                    message: err.message,
+                });
+            }
+        }));
+    }).nThen(function () {
+        //console.log(Env.limits);
+        cb();
+    });
 };
 
 var prepareEnv = function (Env, cb) {
-    Env.customLimits = config.customLimits;
-    Quota.applyCustomLimits(Env);
+    //Quota.applyCustomLimits(Env);
 
     nThen(function (w) {
         /*  Database adaptors
@@ -58,6 +79,10 @@ var prepareEnv = function (Env, cb) {
             }
             Env.blobStore = _;
         }));
+    }).nThen(function (w) {
+        loadPremiumAccounts(Env, w(function (/* err */) {
+            //if (err) { }
+        }));
     }).nThen(function () {
         cb();
     });

server.js

@@ -12,31 +12,10 @@ var Default = require("./lib/defaults");
 var Keys = require("./lib/keys");
 
 var config = require("./lib/load-config");
+var Env = require("./lib/env").create(config);
 
 var app = Express();
 
-// mode can be FRESH (default), DEV, or PACKAGE
-
-var FRESH_KEY = '';
-var FRESH_MODE = true;
-var DEV_MODE = false;
-if (process.env.PACKAGE) {
-// `PACKAGE=1 node server` uses the version string from package.json as the cache string
-    console.log("PACKAGE MODE ENABLED");
-    FRESH_MODE = false;
-    DEV_MODE = false;
-} else if (process.env.DEV) {
-// `DEV=1 node server` will use a random cache string on every page reload
-    console.log("DEV MODE ENABLED");
-    FRESH_MODE = false;
-    DEV_MODE = true;
-} else {
-// `FRESH=1 node server` will set a random cache string when the server is launched
-// and use it for the process lifetime or until it is reset from the admin panel
-    console.log("FRESH MODE ENABLED");
-    FRESH_KEY = +new Date();
-}
-
 (function () {
     // you absolutely must provide an 'httpUnsafeOrigin'
     if (typeof(config.httpUnsafeOrigin) !== 'string') {
@@ -64,7 +43,7 @@ if (process.env.PACKAGE) {
             config.httpSafePort = config.httpPort + 1;
         }
 
-        if (DEV_MODE) { return; }
+        if (Env.DEV_MODE) { return; }
         console.log(`
     m     m   mm   mmmmm  mm   m mmmmm  mm   m   mmm    m
     #  #  #   ##   #   "# #"m  #   #    #"m  # m"   "   #
@@ -81,15 +60,6 @@ if (process.env.PACKAGE) {
     }
 }());
 
-var configCache = {};
-config.flushCache = function () {
-    configCache = {};
-    FRESH_KEY = +new Date();
-    if (!(DEV_MODE || FRESH_MODE)) { FRESH_MODE = true; }
-    if (!config.log) { return; }
-    config.log.info("UPDATING_FRESH_KEY", FRESH_KEY);
-};
-
 var setHeaders = (function () {
     // load the default http headers unless the admin has provided their own via the config file
     var headers;
@@ -144,6 +114,7 @@ if (!config.logFeedback) { return; }
 
 const logFeedback = function (url) {
     url.replace(/\?(.*?)=/, function (all, fb) {
+        if (!config.log) { return; }
         config.log.feedback(fb, '');
     });
 };
@@ -182,7 +153,7 @@ app.get(mainPagePattern, Express.static(__dirname + '/customize'));
 app.get(mainPagePattern, Express.static(__dirname + '/customize.dist'));
 
 app.use("/blob", Express.static(Path.join(__dirname, (config.blobPath || './blob')), {
-    maxAge: DEV_MODE? "0d": "365d"
+    maxAge: Env.DEV_MODE? "0d": "365d"
 }));
 app.use("/datastore", Express.static(Path.join(__dirname, (config.filePath || './datastore')), {
     maxAge: "0d"
@@ -197,22 +168,10 @@ app.use("/customize.dist", Express.static(__dirname + '/customize.dist'));
 app.use(/^\/[^\/]*$/, Express.static('customize'));
 app.use(/^\/[^\/]*$/, Express.static('customize.dist'));
 
-var admins = [];
-try {
-    admins = (config.adminKeys || []).map(function (k) {
-        // XXX is there any reason not to use Keys.canonicalize ?
-        // return each admin's "unsafeKey"
-        // this might throw and invalidate all the other admin's keys
-        // but we want to get the admin's attention anyway.
-        // breaking everything is a good way to accomplish that.
-        return Keys.parseUser(k).pubkey;
-    });
-} catch (e) { console.error("Can't parse admin keys"); }
-
 var serveConfig = (function () {
     // if dev mode: never cache
     var cacheString = function () {
-        return (FRESH_KEY? '-' + FRESH_KEY: '') + (DEV_MODE? '-' + (+new Date()): '');
+        return (Env.FRESH_KEY? '-' + Env.FRESH_KEY: '') + (Env.DEV_MODE? '-' + (+new Date()): '');
     };
 
     var template = function (host) {
@@ -227,12 +186,12 @@ var serveConfig = (function () {
                 allowSubscriptions: (config.allowSubscriptions === true),
                 websocketPath: config.externalWebsocketURL,
                 httpUnsafeOrigin: config.httpUnsafeOrigin,
-                adminEmail: config.adminEmail, // XXX mutable
-                adminKeys: admins,
-                inactiveTime: config.inactiveTime, // XXX mutable
-                supportMailbox: config.supportMailboxPublicKey,
-                maxUploadSize: config.maxUploadSize, // XXX mutable
-                premiumUploadSize: config.premiumUploadSize, // XXX mutable
+                adminEmail: Env.adminEmail,
+                adminKeys: Env.admins,
+                inactiveTime: Env.inactiveTime,
+                supportMailbox: Env.supportMailboxPublicKey,
+                maxUploadSize: Env.maxUploadSize,
+                premiumUploadSize: Env.premiumUploadSize,
             }, null, '\t'),
             'obj.httpSafeOrigin = ' + (function () {
                 if (config.httpSafeOrigin) { return '"' + config.httpSafeOrigin + '"'; }
@@ -253,28 +212,29 @@ var serveConfig = (function () {
         var host = req.headers.host.replace(/\:[0-9]+/, '');
         res.setHeader('Content-Type', 'text/javascript');
         // don't cache anything if you're in dev mode
-        if (DEV_MODE) {
+        if (Env.DEV_MODE) {
             return void res.send(template(host));
         }
         // generate a lookup key for the cache
         var cacheKey = host + ':' + cacheString();
 
-        // XXX we must be able to clear the cache when updating any mutable key
+        // FIXME mutable
+        // we must be able to clear the cache when updating any mutable key
         // if there's nothing cached for that key...
-        if (!configCache[cacheKey]) {
+        if (!Env.configCache[cacheKey]) {
             // generate the response and cache it in memory
-            configCache[cacheKey] = template(host);
+            Env.configCache[cacheKey] = template(host);
             // and create a function to conditionally evict cache entries
             // which have not been accessed in the last 20 seconds
             cleanUp[cacheKey] = Util.throttle(function () {
                 delete cleanUp[cacheKey];
-                delete configCache[cacheKey];
+                delete Env.configCache[cacheKey];
             }, 20000);
         }
 
         // successive calls to this function
         cleanUp[cacheKey]();
-        return void res.send(configCache[cacheKey]);
+        return void res.send(Env.configCache[cacheKey]);
     };
 }());
 
@@ -297,7 +257,7 @@ app.use(function (req, res, next) {
     send404(res, custom_four04_path);
 });
 
-var httpServer = Http.createServer(app);
+var httpServer = Env.httpServer = Http.createServer(app);
 
 nThen(function (w) {
     Fs.exists(__dirname + "/customize", w(function (e) {
@@ -323,11 +283,12 @@ nThen(function (w) {
 
     // Initialize logging then start the API server
     require("./lib/log").create(config, function (_log) {
+        Env.Log = _log;
         config.log = _log;
-        config.httpServer = httpServer;
 
         if (config.externalWebsocketURL) { return; }
-        require("./lib/api").create(config);
+
+        require("./lib/api").create(Env);
     });
 });
 

www/admin/app-admin.less

@@ -35,27 +35,26 @@
         code {
             cursor: pointer;
         }
-        ul.cp-compact {
-            .cp-admin-limit {
-                display: flex;
-                align-items: center;
-                overflow: hidden;
+        table {
+            td:not(:last-child) {
+                padding-right: 20px;
                 white-space: nowrap;
+            }
+            td:last-child {
+                min-width: 0;
+                white-space: nowrap;
+                overflow: hidden;
                 text-overflow: ellipsis;
-                ul {
-                    display: inline;
-                    overflow: hidden;
-                    white-space: nowrap;
-                    text-overflow: ellipsis;
-                    li {
-                        display: inline-block;
-                        &.limit {
-                            width: 130px;
-                        }
-                        &.plan {
-                            width: 140px;
-                        }
-                    }
+                max-width: 500px;
+            }
+            @media screen and (max-width: 1200px) {
+                td.note {
+                    display: none;
+                }
+            }
+            @media screen and (max-width: 1400px) {
+                td.plan {
+                    display: none;
                 }
             }
         }

www/admin/inner.js

@@ -43,7 +43,7 @@ define([
         'general': [
             'cp-admin-flush-cache',
             'cp-admin-update-limit',
-            'cp-admin-registration'
+            // 'cp-admin-registration',
         ],
         'quota': [
             'cp-admin-defaultlimit',
@@ -107,10 +107,6 @@ define([
         });
         return $div;
     };
-    Messages.admin_registrationHint = "Restrict registration..."; // XXX
-    Messages.admin_registrationTitle = "Restrict registration"; // XXX
-    Messages.admin_registrationButton = "Restrict"; // XXX
-    Messages.admin_registrationAllow = "Allow"; // XXX
     create['registration'] = function () {
         var key = 'registration';
         var $div = makeBlock(key, true);
@@ -153,11 +149,6 @@ define([
                              : Messages._getKey('formattedMB', [value]);
     };
 
-    Messages.admin_defaultlimitTitle = "Storage limit"; // XXX
-    Messages.admin_defaultlimitHint = "Maximum storage limit per user drive and team drive when no custom rule is applied"; // XXX
-    Messages.admin_defaultlimitTitle = "New default limit (MB)"; // XXX
-    Messages.admin_setlimitButton = "Set limit"; // XXX
-    Messages.admin_limit = "Current default limit: {0}";
     create['defaultlimit'] = function () {
         var key = 'defaultlimit';
         var $div = makeBlock(key);
@@ -198,10 +189,6 @@ define([
         });
         return $div;
     };
-    Messages.admin_getlimitsHint = "List all the custom storage limits applied to your instance."; // XXX
-    Messages.admin_getlimitsTitle = "Custom limits"; // XXX
-    Messages.admin_limitPlan = "Plan: {0}";
-    Messages.admin_limitNote = "Note: {0}";
     create['getlimits'] = function () {
         var key = 'getlimits';
         var $div = makeBlock(key);
@@ -223,8 +210,7 @@ define([
                     return obj[a].limit > obj[b].limit;
                 });
 
-                var addClass = "";
-                if (list.length > 10) { addClass = ".cp-compact"; }
+                var compact = list.length > 10;
 
                 var content = list.map(function (key) {
                     var user = obj[key];
@@ -236,40 +222,44 @@ define([
                     var keyEl = h('code.cp-limit-key', key);
                     $(keyEl).click(function () {
                         $('.cp-admin-setlimit-form').find('.cp-setlimit-key').val(key);
+                        $('.cp-admin-setlimit-form').find('.cp-setlimit-quota').val(Math.floor(user.limit/1024));
+                        $('.cp-admin-setlimit-form').find('.cp-setlimit-note').val(user.note);
                     });
-                    return h('li.cp-admin-limit', {
-                        title: addClass ? title : ''
-                    }, [
+                    if (compact) {
+                        return h('tr.cp-admin-limit', {
+                            title: title
+                        }, [
+                            h('td', keyEl),
+                            h('td.limit', Messages._getKey('admin_limit', [limit])),
+                            h('td.plan', Messages._getKey('admin_limitPlan', [user.plan])),
+                            h('td.note', Messages._getKey('admin_limitNote', [user.note]))
+                        ]);
+                    }
+                    return h('li.cp-admin-limit', [
                         keyEl,
                         h('ul.cp-limit-data', [
                             h('li.limit', Messages._getKey('admin_limit', [limit])),
-                            //h('li.plan', Messages._getKey('admin_limitPlan', [user.plan])),
+                            h('li.plan', Messages._getKey('admin_limitPlan', [user.plan])),
                             h('li.note', Messages._getKey('admin_limitNote', [user.note]))
                         ])
                     ]);
                 });
-                $div.append(h('ul.cp-admin-all-limits'+addClass, content));
+                if (compact) { return $div.append(h('table.cp-admin-all-limits', content)); }
+                $div.append(h('ul.cp-admin-all-limits', content));
             });
         };
         APP.refreshLimits();
         return $div;
     };
 
-    Messages.admin_setlimitHint = "Get the public key of a user and give them a custom storage limit. You can update an existing limit or remove the custom limit."; // XXX
-    Messages.admin_setlimitTitle = "Apply a custom limit"; // XXX
-    Messages.admin_limitUser = "User's public key"; // XXX
-    Messages.admin_limitMB = "Limit (in MB)"; // XXX
-    Messages.admin_limitSetNote = "Custom Note"; // XXX
-    Messages.admin_invalKey = "Invalid public key";
-    Messages.admin_invalLimit = "Invalid limit value";
     create['setlimit'] = function () {
         var key = 'setlimit';
         var $div = makeBlock(key);
 
         var user = h('input.cp-setlimit-key');
         var $key = $(user);
-        var limit = h('input', {type: 'number', min: 0, value: 0});
-        var note = h('input');
+        var limit = h('input.cp-setlimit-quota', {type: 'number', min: 0, value: 0});
+        var note = h('input.cp-setlimit-note');
         var remove = h('button.btn.btn-danger', Messages.fc_remove);
         var set = h('button.btn.btn-primary', Messages.admin_setlimitButton);
         var form = h('div.cp-admin-setlimit-form', [
@@ -620,7 +610,6 @@ define([
             active = active.split('-')[0];
         }
         common.setHash(active);
-        Messages.admin_cat_quota = 'Quotas'; // XXX
         Object.keys(categories).forEach(function (key) {
             var $category = $('<div>', {'class': 'cp-sidebarlayout-category'}).appendTo($categories);
             if (key === 'general') { $category.append($('<span>', {'class': 'fa fa-user-o'})); }

www/common/onlyoffice/inner.js

@@ -2067,7 +2067,7 @@ define([
                 Title.updateTitle(Title.defaultTitle);
             }
 
-            var version = 'v2a/';
+            var version = 'v2b/';
             var msg;
             // Old version detected: use the old OO and start the migration if we can
             if (privateData.ooForceVersion) {