Plugins

[Nutomic]

No description provided.

[phiresky]

I think generally since the code will be linked into lemmy, any plugin code is automatically AGPL. This means that an instance that has custom plugins must (theoretically) share the code for those plugins with its users. This should probably be mentioned in the docs (and here).

This is similar to Blender, where Blender having a GPL license is often confusing to plugin authors, who usually sell their plugins (but then also need to provide the code publicly)

[Nutomic]

Opened an issue about this: extism/extism#804

[Nutomic]

The answer from extism was also very vague, so I assume that we as maintainers can decide which licensing restrictions apply to plugins. Ive added a section specifying that plugins can use any license including proprietary, otherwise it would not be possible to integrate with proprietary libraries like ios/android push notifications.

[Fmstrat]

Aren't most FCM libraries MIT? Proprietary licenses on API libraries would create havok with enterprise adoption.

While I agree that extism's response says to allow anything (that is my understanding as well, and while I'm also not a lawyer, I have helped author licenses), I personally prefer the Blender method.

Lemmy could get trapped into a closed-source ecosystem if one plugin gets really popular. For instance, what if BlueSky builds a proprietary pre-compiled connector and a large instance begins using it, and thus all others follow?

Perhaps this is OK by maintainers since it is the spirit of free and open, but myself, an N of one, isn't a fan.

[voxpelli]

Elaborated on the licensing: extism/extism#804 (comment)

Any GPL 3.0 compatible code can be included in a plugin, see: https://www.gnu.org/licenses/license-list.html#GPLCompatibleLicenses

And any GPL 3.0 code (and "GPL 2.0 and later" code can be treated as such) can be included in an AGPL 3 license (though plain GPL 2.0 can't), see: https://www.gnu.org/licenses/gpl-3.0.html#section13 / https://www.gnu.org/licenses/agpl-3.0.html#section13

I don't think a plugin will be able to use code under an incompatible license, a non-free license or a proprietary license and I think the plugin will have to be licensed as AGPL 3.0 itself (or maybe as GPL 3.0)

[voxpelli]

This is similar to Blender, where Blender having a GPL license is often confusing to plugin authors, who usually sell their plugins (but then also need to provide the code publicly)

@phiresky The difference here is AGPL vs GPL. They both allow selling the code but they differ in when the source code needs to be made available – AGPL requiring anyone interacting with the software over a network to be offered the source code.

People buying a Blender plugin are legally allowed to request the source code and to spread it, but not everyone that uses what they build using Blender.

Anyone running a Lemmy server needs to offer all the code of their Lemmy server to anyone requesting it.

[bkil]

This is why for a similarly modularized piece of software, we have chosen LGPL so modules can be either free or proprietary as the integrator sees fit. Then it's sufficient to provide the object code of the proprietary library as long as it can be ran/linked after making your changes to the published source in FOSS modules or the FOSS core and recompiling the whole thing.

[Nutomic]

It seems you are right, the SDKs for push notifications which I found (Google Firebase and Microsoft Azure) are under Apache license. So I changed the section to allow any OSI-approved open source license.

[dullbananas]

Something that might be a problem is plugins and their dependencies can't be GPL unless they use the same GPL version as Lemmy, since Lemmy doesn't specify "or later" (LemmyNet/lemmy#5133). Unless plugins are somehow implemented in a way that GPL doesn't interpret as linking.

[voxpelli]

@dullbananas The general conclusion by most others than me here seems to be that there are no such GPL limitations for plugins :/

[phiresky]

Theoretically it would be great if it was possible to externalize core code into plugins - both for separation of concerns, as well as to ensure the plugin structure is performant and complete. Practically it might be too much effort and for many things impossible

[dessalines]

Once people start writing plugins, I could see it going in the opposite direction. A plugin becomes so popular that most instances use it, that we realize its an important enough feature that we should add it into lemmy core. That way everyone could use it without having to host a variety of plugins.

[Nutomic]

Also makes sense, added.

[Fmstrat]

Any thoughts on how login plugins would work under this? For instance OIDC connectors (with overriding server usernames).

I admit I'm slightly biased here being on the FedID.me team and trying to bring federated and distributed credentials into existence ;)

[Nutomic]

It depends what those plugins need to do exactly. If they need extra database tables and api endpoints like LemmyNet/lemmy#4881 then it would be quite complicated. I definitely want to leave such things out of the initial implementation, and keep it simple for now.

[Fmstrat]

Hmm, no API endpoints per-se, but likely something DB related to connect the local username to the OpenID identifier, similar to how Mastodon does it. (We just started experimenting with a FedID implementation with Mastodon via OIDC now.)

It's possible this information could be saved externally for Lemmy, so I'll start considering that.

[phiresky]

maybe you should reference that plugin example code that you wrote a while back in some draft PR?

[phiresky]

Something that I think is missing is constraints on handling of hooks. We need to be careful that if we have blocking hooks we must expect that plugin writers will want to write very expensive/slow calls (including db + network access) into the hooks, and lemmy waits indefinitely for them to complete that might cause issues.

So probably we want the hook registration to enforce both specifying a blocking: true flag as well as a timeout: xxx flag or similar. Restricting hooks to be pure synchronous functions would probably be too restrictiive.

[phiresky]

maybe you should reference that plugin example code that you wrote a while back in some draft PR?

[phiresky]

To be honest - I'm not sure if Extism is the right choice in general. I just read up on it. It seems very neat, with supporting multiple languages and compiling everything to wasm. But I think in the real world, people are going to be disappointed:

Since it compiles things to wasm, things that are pure computations are going to be very neat, but anything that does external disk, network, or other IO will be difficult. For example, wrt JS plugins:

Warning: This is a very bare-bones runtime. It's only for running pure JS code and it does not expose node APIs or the browser APIs. We have limited support for some W3C APIs (e.g. we support Text{Encoder,Decoder} but not fetch), but many modules you take from npm will not work out of the box. There is no support for node APIs or anything that makes syscalls typically. You'll need to polyfill any APIs with a pure JS implementation, which is often possible, but some things, such as controlling sockets, are not possible. Feel free to file an issue if you think an API should be supported though.

So you can't actually import and npm libs, or do fetch, or access a database, or similar.

It might be better to go for some other solution that is maybe more restrictive in which languages it supports (e.g. only JS), but runs in a real JS runtime instead with all power you'd expect.

Of course, if we provide extensive host functions for network access, db access, disk access, etc, it might work, but that would be a lot of effort to maintain and please everyone I think.

[Nutomic]

maybe you should reference that plugin example code that you wrote a while back in some draft PR?

Added the plugin code as an example.

Something that I think is missing is constraints on handling of hooks. We need to be careful that if we have blocking hooks we must expect that plugin writers will want to write very expensive/slow calls (including db + network access) into the hooks, and lemmy waits indefinitely for them to complete that might cause issues.

So probably we want the hook registration to enforce both specifying a blocking: true flag as well as a timeout: xxx flag or similar. Restricting hooks to be pure synchronous functions would probably be too restrictiive.

This is out of scope because its too complicated. Plugin devs and instance admins need to ensure that performance is adequate. Mentioned this under drawbacks.

So you can't actually import and npm libs, or do fetch, or access a database, or similar.

It might be better to go for some other solution that is maybe more restrictive in which languages it supports (e.g. only JS), but runs in a real JS runtime instead with all power you'd expect.

Of course, if we provide extensive host functions for network access, db access, disk access, etc, it might work, but that would be a lot of effort to maintain and please everyone I think.

This seems to be a limitation specifically for Javascript plugins, the plugin development kits for other languages such as Go dont mention any such limitations. Besides all the languages including JS have examples for making HTTP calls from plugins. Database access has to go through Lemmy anyway, to ensure plugins work with different Lemmy versions.

I also looked for different plugin frameworks for Rust and there is not much. There are a few toy projects to run JS from Rust, but making them production ready would take an unreasonable amount of work compared to an major, established project like Extism. The only alternative which looks viable is mlua.

[phiresky]

This seems to be a limitation specifically for Javascript plugins

I'd phrase this differently: The plugin code is compiled to a fully isolated WASM module. The only external access it can do is (a) through host functions we provide, (b) through the "pdk" extism provides. The Pdk provides a limited set of custom methods to do e.g. HTTP requests (for JS as well), but this is very limited (see full list of pdk methods in e.g. Rust or Go), basically just HTTP and logging.

Any stdlib functions in each language don't work, at least as soon as they do any syscalls (http, file system, ...). This also means most libraries in each language won't work.

I also looked for different plugin frameworks for Rust and there is not much.

I think the best alternative might be using deno (deno_runtime crate) directly, they have some blog posts about it, and here is some example code: https://github.com/carloslfu/make-your-own-js-runtime

It's not marketed as a "plugin framework", but it has integrated methods for function calls in both directions: https://docs.rs/deno_core/latest/deno_core/struct.JsRuntime.html

Deno itself is very solid, established, and actively developed, though mainly as a standalone runtime and not for library use.

[Nutomic]

Yes plugins are sandboxed by default, but there is an option to allow network requests or disk access by enabling WASI.

Deno like you say is not a plugin framework, so it would take a lot of extra work from our side to make this work. All of that is provided for free by extism.

[wiki-me]

Maybe add wordpress plugins or discourse plugins to the prior art?

Also maybe mention giving people the ability to experiment and then benchmark new systems (in the motivation or rational sections). quoting linus law of trail and error:

“Don't ever make the mistake [of thinking] that you can design something better than what you get from ruthless massively parallel trial-and-error with a feedback cycle. That's giving your intelligence much too much credit.”

[dessalines]

Once people start writing plugins, I could see it going in the opposite direction. A plugin becomes so popular that most instances use it, that we realize its an important enough feature that we should add it into lemmy core. That way everyone could use it without having to host a variety of plugins.

[phiresky]

Deno like you say is not a plugin framework, so it would take a lot of extra work from our side to make this work

It's really not, though. Look at this official example code: https://github.com/denoland/deno/blob/main/runtime/examples/extension/main.rs It's less than 100 lines to get it running.

there is an option to allow network requests or disk access by enabling WASI.

Maybe, but I have not seen any documentation anywhere on if and how well this works and if it actually means that/which syscalls work. the example only mention their custom HTTP call which is part of the PDK, not wasi.

[voxpelli]

I don't think this is a correct interpretation of AGPL and as such would better be left out of this rather than possibly be in conflict with the AGPL license.

Either that or get the FSF or a lawyer to comment on it.

[voxpelli]

The original discussion: #8 (comment)

[Nutomic]

In what way do you think its not correct?

[voxpelli]

@Nutomic In the ways that the GPL license FAQ describes it, see my elaboration in extism/extism#804 (comment) :)

[voxpelli]

Perhaps @tieguy can help with some guidance?

[Nutomic]

Sorry I missed that comment. The plugins described in this RFC will use the same data structures as our API. And API clients are obviously not bound to GPL limitations. So I think this is similar to the proxy-wasm explanation that you linked.

[voxpelli]

It does sound like it matches with this though:

If the main program dynamically links plug-ins, and they make function calls to each other and share data structures, we believe they form a single combined program, which must be treated as an extension of both the main program and the plug-ins.

[Nutomic]

The data structures are not shared specifically with plugins, but with any code that uses the Lemmy API. Besides I removed the section about "Host Functions", and instead changed it so that plugins can interact with the Lemmy API. So to me this clearly falls under the borderline case where "communication between them is limited to invoking the ‘main’ function of the plug-in with some options and waiting for it to return", meaning it doesnt fall under the license restrictions.

[voxpelli]

Borderline case means that it’s not a clear cut case though? So the wording in this RFC shouldn’t give the impression that it is a clear cut case of it being allowed, especially if it can’t be backed up by a reference to someone knowledgeable in the peculiarities of the AGPL / GPL?

[bkil]

I suggest we define the concept of plugin metadata from the start. Each plugin could be annotated by one or more of the following fields, inspired by apt (see also: other package managers):

• an ID (using Java-like reverse DNS namespace notation)
• a short name that fits a list
• an icon
• a short, few line description of what it does, possibly including indexed keywords (hashtags) if that's not present in a designated field
• an easily parsable version number
• copyright declaration (license SPDX or link, authors, date)
• link to main landing page of own website
• link to end user or sysadmin documentation (handbook, wiki, etc)
• link to source code
• link to a bug tracker, support contact channel or group, also how to report security issues
• an exhaustive list of all API hooks the plugins would like permission to access during its operation (both to allow the admin to review it before installation and as defense in depth against bugs)
• a list of plugins (possibly with minimum & maximum version number) which are required, suggested, conflicting and replaced by this one respectively

[Nutomic]

Added a note about metadata near the beginning under File Structure. Though I want to keep it simple with only a few fields.