RFC: The plugin system


#1

One of the key advantage of an open-source software is that it empowers interested developers to read and understand the source code, and extend it with their imagination as the only limit. On the matter I have to say that I have been pleasantly surprised by the quality and the value brought by some of the recent contributions to Wekan.

However contributing to Wekan still has a high barrier to entry in that you have to implement something that every single user will want to run. To remove that constraint I talked publicly on many occasions about the need to transform Wekan from a monolithic application to a customizable set of plugins on top of an open-core. This document provides more details about this idea.

Many other softwares have proven to be successful with a “micro-kernel” approach, but unfortunately not so much of the end-user softwares. The Wordpress publishing platform and the Atom code editor (links point to their respective plugin marketplace) are two good counter-examples that fit in the “modular end-user software” category. The below proposal will take inspiration on these.

Desirable plugins

Let’s set some examples of the kind of Wekan extensions that our plugin system should allow:

  • Cards synchronization with a third party service. The most popular case of these being synchronization with GitHub/Gitlab as proposed for instance by Waffle or ZenHub for managing your issues and pull requests using a Kanban view. Other desirable synchronization would happen with tools like Saleforce, or even Gmail as proposed by Sortd.
  • Data import and export. This is sort of a lighter linkage than the previous category, here we don’t want the full synchronization but just to be able to process one-time import or export—for instance of a Microsoft Excel .xls file. Our existing Trello importation also fits in this category, as well as some less popular formats like todo.txt.
  • New data visualization. Even if the Kanban view is at the core of Wekan (after all the “kan” stands for Kanban), a lot of people are interested in different visualizations of their Cards/Tasks. Trello for instance provide a Calendar view where the cards are positioned on their due date. Some third parties implements more complex visualization like a Grantt chart (Elegantt had to develop a Chrome extension, which is more fragile and less integrated than our desired plugin system).
  • New meta-data. Plugins may want to attach some additional data to existing boards and cards. Examples of these would be prioritization plugin allowing you to state “this card is urgent” or “this card has a low priority”, or a plugin to vote on cards for instance for a public feature list.
  • UI modifications. Either for implementing new features, buttons, popups or any UI element, or to change Wekan theme for instance to integrate it with an existing website by adding a menu bar on top of the screen.

These examples fully shows the need for a plugin system as they are both desirable for some users (and most of the time actually requested by them) and too specific to force their usage to everyone.

Requirements

Telescope is another Meteor application that shares our need for modularity and decided to implement its plugin as Meteor packages. This mean that you can add a plugin by using meteor add and remove it with meteor remove and then you have to rebuilt your application and start it. Discourse the software that runs the Wekan forum has a similar model of plugins as Ruby gems. This is a fine choice for their target audience.

This is however not suitable for us, as we don’t want to force our end users to use a CLI build tool, and this is where the Atom plugin system is a good examples that matches our goals:

  • Plugins can be downloaded from a central repository with an App-store like browsing experience
  • Plugins are hot swappable, which mean that you have neither to rebuild nor to restart the application in order to start or stop a given plugin
  • Plugins can have some dependencies, along with the obvious advantages of an explicit dependencies management, this allows the creation of “packs” as suggested in https://discuss.wekan.io/t/wekan-plugin-system/78, which are just sets of plugins that work well with each others.

Distribution

Plugins systems share a lot of common characteristics that we don’t want to re-invent, things like packaging, versionning, or dependency arbitrations should be considered as solved problems. Consequently we will build our plugin system on top of an existing package system and infrastructure. And as we are a JavaScript application our candidate of choice is NPM.

Meteor packages were an interesting basis as well since the “isobuild” format they use has the advantage of a clear comprehension of our full stack architecture, which means that it is easy for a package to define “this file goes on the server” and “this one on the mobile app”. However as outlined by the discussions happening in the Meteor community, the advantages of using the same tools as the broader JavaScript community generally outweighs the better integration provided by the custom development of these tools. Nowadays, NPM works perfectly on the client, and its rich ecosystem makes it the best candidate for the job. It’s also worth noting that the Atom package manager is a light wrapper on top of NPM.

So Wekan plugins will be distributed as NPM packages. The weak/flexible package.json manifest will be useful to specify additional meta-data that our plugins will require such as entry points for the different target platforms (server, client browser, mobile app) or category tags (“theme”, “productivity”, etc.). Plugins will be installed in a directory of the host (or container in the case of a Docker installation) . The exact location is yet to be decided but could be /var/lib/wekan/plugins/ on Linux and MacOS (and probably something like C:\Program Files\Wekan\plugins\ on Windows).

We shall also keep a registry of installed packages with their meta-data (in particular the version number) in the database so that we could re-install them if someone restore a Wekan instance from a database dump (the promise is that if you backup your database and restore it on another host you’ll end up with the exact same application state). This database collection will also be useful to display an Admin UI to see the various plugins installed and their respective authors, versions, etc.

Note that on Sandstorm, we’ll have to adapt the distribution mechanism. Basically once Sandstorm supports grain plugins, the application will not have to manage anything to handle the distribution and will just see the actual plugin code mounted in some directory in the grain. The system presented in this document should be compatible with the expected future implementation on Sandstorm.

API

Once we have installed a plugin and are able to execute its code, we need to define the interface that it can use to implement the desirable features listed above (like adding meta-data, modifying the UI, etc.).

One important principle at play here is that from a security perspective we won’t try to restrict what a plugin can do. The idea is that installing a plugin implies that you trust it to execute some Turing complete code that could do anything imaginable on the host. While interesting from a engineering perspective, implementing a security isolation with a permission system is out of scope in the short and medium terms. The goal of the plugin APIs is only to provide some clean canonical way, that won’t possibly break with the future evolutions of the application, to implement desirable features and not to prevent certain type of modifications such as monkey-patching core primitives or sending data to an attacker server.

Plugins will use ES6 modules to import and export JavaScript objects. Using the modern standard is an obvious choice here, especially as we choose NPM as the distribution mechanism, and because it will allow some static analysis on the dependency graph in the future. So for example if a plugin needs to access the Boards and Cards collections it will:

import { Boards, Cards } from 'wekan/models';

Wekan core objects will be available under the wekan namespace, for instance wekan/notifications or wekan/components. Wekan plugins shall also be registered under this namespace, for instance wekan/slack-notifications or wekan/github-sync. Meteor objects will be available under the meteor namespace (per 1.3-beta3 implementation), for example meteor/tracker.

Storing data

Like the Wekan core code, plugins must store their persistent state in (and only in) the Mongo database. They can either store their state in a custom collection:

import { Mongo } from 'meteor/mongo';

export GmailTags = new Mongo.Collection('gmail-tags');

or they can extend the schema of an existing collection to add some meta-data:

import { Cards } from 'wekan/models';

Cards.attachSchema({
  karma: {
    type: Number
  }
});

in which case they may also want to run some migrations and set the default data on insertion:

import { Cards } from 'wekan/models';
import { Migrations } from 'wekan/migrations';

Migrations.add('set-default-karma', () => {
  Cards.update({ karma: { $exists: false }}, {
    $set: {
      karma: 100
    }
  });
});

Cards.before.insert((userId, doc) {
  return { ...doc, karma: 100 };
});

Settings

Plugins will be able to define some settings that the application will expose on the Admin panel, or the board setting view depending on the plugin needs (for instance data important will be configured at the instance level while GitHub synchronization will be set at the board level). These settings will be saved in the plugin meta-data collection. I haven’t investigated the exact API that is needed here, but it should be pretty similar to what iOS applications or Wordpress plugins use. For each setting plugins will have to define a type (Boolean, Select list, Text field, etc.), a description, and a default value.

Of course, as in iOs some plugins will prefer to handle their settings themselves and will display their setting forms in some other specific place in the UI. This is totally fine, as our settings API will just be a recommendation to favor general consistency among plugins developed by different authors.

UI modifications

The last API discussed in this document relates the plugins modifying the UI. This API will be exclusively based on React components, as React core principles allows for a cleaner component composition than the Blaze components that we currently use. Migrating Wekan to React was already one of the goal for 2016 for various reasons exposed here, UI composition through distinct plugins just add one to the list.

Each component will expose some methods for components to register themselves and modify the render tree. For instance the minicard component could have a way to register a new indicators:

import { MiniCard } from 'wekan/components';
import { React } from 'meteor/react';

// Add an indicator to show if the card karma is good or bad
class KarmaIndicator extends React.Component {
  render() {
    const { karma } = this.props;
    let color;
    if (karma > 100) { color = 'green'; }
    else if (karma === 100) { color = 'grey'; }
    else { color = 'red'; }
    return (
      <img src="/icons/karma-${color}.svg" title="Karma score: ${karma}"/>
    );
  }
}

MiniCard.registerIndicator({
  priority: 4,
  component: KarmaIndicator
});

These custom components hooks will be useful for lists or menus components for which it makes sense for a plugin to hook in. But for the general case it’s not easy for the component author to think a priori of a way third parties will want to extend it. Because of that we will have a default composition method that consist of including an existing component into a wrapper:

import { CardDescription } from 'wekan/component';
import { React } from 'meteor/react';

// Show a warning if the card description has more than 100 lines
CardDescription.registerWrapper(function (childComponent) {
  return class cardDescriptionNotTooLong extends React.Component {
    cardDescriptionIsTooLong() {
      const { description } from this.props;

      return description.split(/\r\n|\r|\n/).length > 100;
    }

    render() {
      const warningMessage = this.cardDescriptionIsTooLong() ?
        <p style={{color: 'red'}}>Your description is too long!</p> : null;
      return (
        <div>
          {warningMessage}
          <ChildComponent {...this.props} />
        </div>
      );
    }
  }
});

All core and non-core components will expose this registerWrapper method by default. The actual implementation of this method will rely on a library like Recompose or React-Komposer (by Kadira). When a wrapper is defined on a component it’s actually the wrapper render() function that is called in every places the original wrapped component was called, which allow to extend but also to replace some of the features or UI elements of any React component.

We still need to define how these components are going to declare the data they need to receive in their props. Project like Relay and Om Next have put an emphasis on how powerful it is to have a static declaration of the data needed by a component. These declarations can be composed up to the root of the component tree to generate a request that contains all the needed data of the current view. Unfortunately we don’t use such a composable data layer yet (we will consider GraphQL once Kadira’s Lokka gets support for LiveQueries) and until that we will probably have to rely on some add-hoc hooks into the server side publications.


This document presented a general overview of the goals, principles, and main APIs of my plugin system proposition. The design exposed here is not set in stones and will evolve as we start creating the first few Wekan plugins in the coming weeks. I will love to gets some feedback from existing Wekan contributors and from people who forked the project to modify it for their need but didn’t try to merge their change back into core, or really for anyone potentially interested in a Wekan plugin development.

I also wonder if the two other big open-source Meteor applications, Rocket.Chat and Telescope would be interested in a collaboration on this project and at the very least if they share the goals and motivations exposed in the beginning.


#2

I believe a plugin system will be greatly beneficial to Wekan - and could even be reused (even if only at the specification level) by a lot of applications. I also think your RFC is a great synthesis of the current state of the art, so I only have a few remarks:

Namespaces
It could be beneficial to separate core and plugin (even if only for support reasont), so maybe use wekan-plugin or wekan/plugin as a dedicated namespace for plugins.

Extension points
I agree that plugins should have the ability to execute arbitrary code and plug wherever they want. But I have also witnessed in the Wordpress eco-system the power of having clearly definet places where you can plug.
So I propose to have the ability to define “extension points” (in core or plugins), document them well, and let plugins register for these. Then the code that defines the end points manage its subscribers (the managing code might impose some contract on its subscribers):

  • a menu end point would expect a few methods (label, panelToDisplay,…) and then display the labels of all its registered plugins in its list of items
  • an import endpoint would expect a label and an import method, and would use these to display the new import system among the existing ones
  • a metadata endpoint would let plugins register new metadata types
  • the mechanism for registering and dispatching could be abstracted and shared by all code
  • also, having a unique end point manager could help detect issues (duplicate names, registering on a non-existing extension point)
  • a special case of extension points would be event dispatchers - that way, plugins could hook cleanly into other parts of the application a bit like your UI wrapper

This would be a more general version of what you proposed for UI, and would provide an easier development experience for plugins with limited scope. Full-scale plugins (like a comple UI overhaul) would still be able to bypass it and code against all existing objects.

Lifecycle
An important element of plugins management is their lifecycle, we should define a few standard methods that get called as the plugin is installed / configured / started / stopped / removed

Let us know how we can help with this.


#3

Goods points Xavier!

Namespaces: Yes putting plugins in wekan/plugins might be a good idea. I originally put them in the same namespace than core following what meteor has done with its package system, but an important distinction of their system is that they force non-core packages to be prefixed by the author name, for instance meteor/kadira:flow-router. I imagine that would be an alternative as well, but it isn’t really consistent with NPM conventions.

One other thing I forgot to mention is that NPM packages names should follow a naming convention, probably by being prefixed by wekan-, ie wekan-my-plugin (at least if we use NPM servers for distribution).

Extension points: If only JavaScript has a built-in primitive like an interface or traits to support this model! Maybe in ES2022? One potential candidate for the concrete implementation of these extension points would be Atom services API. Do you have an opinion on this API?

Lifecycle: Yes it’s a good idea. In the karma example I defined a migration to run some code at plugin installation, which is fine in this case I guess, but some clean lifecycle hooks would be a plus.


On the methodology: I will edit my proposal to include the various improvements suggested. I understand that on a forum discussion we are not supposed to edit our messages to change what they say but this case is a bit particular as there is some value to have an up to date general overview document. Also Discourse has a diffing feature so previous versions aren’t lost and people interested can look at what precisely changed since the last time they checked the document.


#4

I like the ideas behind “Atom services API” but it seems to me the implementation is slightly over-engineered, basically re-describing in json what you already have in your code (like listing the public methods you expose, if I understand correctly). And also strangely incomplete, as you describe the name of the method, but not its parameters.

Also, I believe rigorous multi-versioning and dependency management is a very tough problem so unless you have a strong use case (like bazillion people depending on your API not breaking EVER), you should leave it to package managers, and just list you compatiblity (or not) in your release notes.

I would favor a more code-centric view, like exporting only the methods you want to publish (maybe hide the ones that the system requires but should not expose behind naming convention) and adding good complete jsdoc to it, or maybe a few check(), Match, or SimpleSchema definitions.
Then provid a nice register singleton that could basically provide the same info that all the json would have, but without having to keep it in sync, and accessible at runtime.

Like:

  • in core a “PluginRegistry” class/component with methods “registerPlugin”, “listPlugins”, “start(plugin)”…
  • in core an “ExtensionRegistry” with “defineExtensionPoint(name, params, provider)”, “subscribe(extensionPoint, plugin, priority)”, “callSubscribers(name, params)”,…
  • in the plugin, you would call these methods in lifecycle methods (onInit, onRegister,…) or in actions.

#5

Super, but I still can’t quite visualise the architecture:

Core <-> Plugin Admin <-> Plugin install/removal/config/enable/disable
         <-> Plugin API     <-> Plugin code

We need a helloworld example!

I’m happy to help with:

Webhooks example - e.g. github issues = Cards synchronization with a third party service + Data import and export
graphql, json, xml, csv, cron = Data import and export
Pomodoro timer example = New meta-data + New data visualization + UI modifications
Markdown editor example = New meta-data + New data visualization + UI modifications

But there are plugins which should be system wide, some for certain users, some for certain boards, some for lists (if not removed from wekan), some for certain cards, some for certain card labels.

I will give more feedback soon but am really keen to see a hellowekanworld example for the listed desirable plugins, starting with the simplest first, which I guess would be UI modifications and New meta-data - e.g. add a due date and display it.

Please do consider being able to have an optional/separate mongo db for the plugins.
Please also consider being able to enable external plugins - i.e. not on the wekan core installation itself, e.g. plugin hosted services.
Please also consider building the Admin UI as a plugin itself, and not part of core

tbc

h


#6

I think the proposal for extending wekan like this is fantastic. Im not an engineer so will offer some tentative suggestions.First, I think that the ability to house these plugins in disparate repositories and import them with NPM is the right approach. What you may need to do however, is offer some form of validation service/marketplace that can verify that the given plugin will work against version x.xx of Wekan. This might end up taking a lot of time so I’m not sure how you want to run this or how WordPress tackles this issue but worth considering before you run into users frustrated by not knowing ahead of time if one plugin will actually work with their instance of Wekan.

I also think that enabling the plugins to run arbitrary code is a huge plus. However that may mean that you want to be careful who you expose the Admin UI to. It necessarily precludes (I think) the owner of a board from being able to import any kind of plugin, rather these permissions should reside only with the wekan sys admin (so to speak). The sys admin enables a set of plugins to be used by board owners or, perhaps, by specific users in the system.

One item I didnt catch in your proposal is if the extensions themselves could expose outward facing APIs. This could be a way of (for example) enabling external services to leverage some of the extensions or provide opportunities for card specific callbacks, cards to interact with each other, extend functionality through webhooks etc. I don’t know if that is implicit in your proposal (as I said, Im not an engineer!).

Lastly, perhaps it is an idea to start with a subset of the plugin architecture you are suggesting and build that. I would suggest something of high value and reasonably well contained. My vote would be to focus on basic extensions to cards. Start with a basic card boilerplate and some simple example code showing how to extend it eg. add new UI elements, and store and retrieve data, hit an external service and retrieve and display its payload, alter values in other cards. This could already be used creatively in many ways and would hold a lot of value for people.You might then see some uptake and then push forward the plugins into other areas of the wekan ecosystem.


#9

Yes

Yes, that arises naturally from relying on ES6 modules under the hood, so a plugin A could export a methodA and then a plugin B could import { methodA } from 'wekan/a'.


#10

Is there any codding done for this specification? Back in January work was expected to start in few weeks.


#11

@mquandalle are you in touch with Rocket.Chat? They want to build a plugin system also.
I have no idea if my intervention is relevant, but would be nice if this issue is solved for meteor community (or at least npm). But I’m not sure that it is feasible.

I think the easiest would be to have npm plugins with a UI for the admin to manage. But I’m not sure it is relevant, just discard my comment if it is not relevant :smile:


#12

Hey @pierreozoux, I’m unaware of Rocket.Chat plan is regards to a plugin system. Could you point me to some public discussion about it? Also cc @engelgabriel.


#13

And here you are:

https://github.com/RocketChat/Rocket.Chat/issues/1859

Hope it helps :smile:


#14

I agree that NPM based packaging would be a good approach.


#15

We are not going to do NPM-based plugin system because of issues mentioned here:


#16

I don’t see any issues for NPM plugin system mentioned in that issue. Care to link to those directly?


#17

Themes issue: Implementation plan v2, Pro 1st and 2nd point: unable to install npm package plugins to Docker container and read-only Sandstorm filesystem while Wekan is running. That’s why no plugin system planned.


#18

Makes sense for Docker and Sandstorm. Thanks for pointing out!