Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Plugins #8

Open
wants to merge 26 commits into
base: main
Choose a base branch
from
Open

Plugins #8

Changes from 4 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
1f2b50b
Plugins
Nutomic Dec 2, 2024
4b88bf5
Update 0000-plugins.md
Nutomic Dec 2, 2024
d1acc83
Update 0000-plugins.md
Nutomic Dec 2, 2024
18dc550
Update 0000-plugins.md
Nutomic Dec 2, 2024
31b17e3
add example, more details
Nutomic Dec 3, 2024
c58c536
some changes
Nutomic Dec 4, 2024
8d645e4
licensing, future possibilities
Nutomic Dec 9, 2024
7b54dbf
Add prior art
Nutomic Dec 12, 2024
cbca68c
change licensing
Nutomic Dec 12, 2024
686d0b9
plugin loading
Nutomic Dec 12, 2024
ae503df
summary/motivation
Nutomic Dec 17, 2024
73cbcfd
fix go
Nutomic Dec 17, 2024
5963ecb
metadata
Nutomic Dec 17, 2024
22e4aef
get rid of host functions, rewrite drawbacks
Nutomic Dec 18, 2024
208ca51
various changes
Nutomic Dec 18, 2024
cbf1293
fix metadata
Nutomic Mar 11, 2025
8191727
Update file structure based on impl
Nutomic Mar 13, 2025
47d4e07
fix json
Nutomic Mar 13, 2025
1ab9906
update golang code
Nutomic Mar 13, 2025
36a2e4f
change example plugin to work on post title
Nutomic Mar 14, 2025
2686c6b
add private message hooks
Nutomic Mar 17, 2025
ca648b9
document lemmy_url config
Nutomic Mar 21, 2025
8f3f589
add more plugin hooks, better names
Nutomic Mar 21, 2025
cbc1783
add link to plugins repo, remove initial plugins section
Nutomic Mar 21, 2025
1939905
revert fmt
Nutomic Mar 21, 2025
8f3a62f
fix hook
Nutomic Mar 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 98 additions & 0 deletions 0000-plugins.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
- Feature Name: plugins
- Start Date: 2024-12-02
- RFC PR: [LemmyNet/rfcs#0000](https://github.com/LemmyNet/rfcs/pull/0008)
- Lemmy Issue: [LemmyNet/lemmy#0000](https://github.com/LemmyNet/lemmy/issues/3562)

# Summary

Plugins will make Lemmy more flexible, and allow implementing features which are too niche for merging into Lemmy core. It will allow Lemmy to focus on core features, while allowing outside developers to contribute new features without using Rust.

# Motivation

TODO

# Guide-level explanation

## Extism

An open source framework to develop plugins for Rust projects. It supports numerous different languages by compiling them to webassembly which gets called from Rust.

https://extism.org/

# Reference-level explanation

## Plugin Hooks

Lemmy will have to add hooks for specific actions, which can then be used by plugins. In general there are two types of hooks: before an action is written to the database, so it can be rejected or altered. And after it is written to the database, mainly for different types of notifications.

For the initial implementation, the following hooks will be available:

### Before writing to Database

These hooks can be used to reject or alter user actions based on different criteria.

- Post
- Create local post
- Update local post
- Receive federated post
- Comment
- Create local comment
- Update local comment
- Receive federated comment
- New vote

### After writing to Database

These are mainly useful to generate notifications.

- Post created or edited
- Comment created or edited
- New vote

## Host Functions

Lemmy database functions can be exposed by plugins via [host functions](https://extism.org/docs/concepts/host-functions). For the beginning the following functions will be exposed:

- Instance::read_or_create

More functions may be added if they are needed for the initial plugins.

## Initial Plugins

These can mainly serve as proof of concept, and as examples which can be modified by plugin developers. Include integration tests to ensure that functionality works with new Lemmy changes. These plugins should be written in a variety of languages. It could make sense to put them into a separate repo.

- Block only downvotes from a specific instance
- Replace words in post or comment text (eg "cloud to butt")
- Replace Youtube URLs in post links with [Invidious](https://invidious.io/)
- Push notifications (send data of new posts to localhost url, [#2631](https://github.com/LemmyNet/lemmy/issues/2631))
- Language detection (automatically apply language tag to posts/comments, [#2870](https://github.com/LemmyNet/lemmy/issues/2870))
- Additional suggestions welcome

# Drawbacks

Plugins will likely have worse performance than native Rust.

# Rationale and alternatives

TODO

# Prior art

TODO

# Unresolved questions

What are the licensing requirements for plugins, if any? Are proprietary plugins allowed or do they need to be under AGPL like Lemmy? May have to ask Extism maintainers about legal conditions.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think generally since the code will be linked into lemmy, any plugin code is automatically AGPL. This means that an instance that has custom plugins must (theoretically) share the code for those plugins with its users. This should probably be mentioned in the docs (and here).

This is similar to Blender, where Blender having a GPL license is often confusing to plugin authors, who usually sell their plugins (but then also need to provide the code publicly)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened an issue about this: extism/extism#804

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The answer from extism was also very vague, so I assume that we as maintainers can decide which licensing restrictions apply to plugins. Ive added a section specifying that plugins can use any license including proprietary, otherwise it would not be possible to integrate with proprietary libraries like ios/android push notifications.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aren't most FCM libraries MIT? Proprietary licenses on API libraries would create havok with enterprise adoption.

While I agree that extism's response says to allow anything (that is my understanding as well, and while I'm also not a lawyer, I have helped author licenses), I personally prefer the Blender method.

Lemmy could get trapped into a closed-source ecosystem if one plugin gets really popular. For instance, what if BlueSky builds a proprietary pre-compiled connector and a large instance begins using it, and thus all others follow?

Perhaps this is OK by maintainers since it is the spirit of free and open, but myself, an N of one, isn't a fan.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Elaborated on the licensing: extism/extism#804 (comment)

Any GPL 3.0 compatible code can be included in a plugin, see: https://www.gnu.org/licenses/license-list.html#GPLCompatibleLicenses

And any GPL 3.0 code (and "GPL 2.0 and later" code can be treated as such) can be included in an AGPL 3 license (though plain GPL 2.0 can't), see: https://www.gnu.org/licenses/gpl-3.0.html#section13 / https://www.gnu.org/licenses/agpl-3.0.html#section13

I don't think a plugin will be able to use code under an incompatible license, a non-free license or a proprietary license and I think the plugin will have to be licensed as AGPL 3.0 itself (or maybe as GPL 3.0)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is similar to Blender, where Blender having a GPL license is often confusing to plugin authors, who usually sell their plugins (but then also need to provide the code publicly)

@phiresky The difference here is AGPL vs GPL. They both allow selling the code but they differ in when the source code needs to be made available – AGPL requiring anyone interacting with the software over a network to be offered the source code.

People buying a Blender plugin are legally allowed to request the source code and to spread it, but not everyone that uses what they build using Blender.

Anyone running a Lemmy server needs to offer all the code of their Lemmy server to anyone requesting it.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is why for a similarly modularized piece of software, we have chosen LGPL so modules can be either free or proprietary as the integrator sees fit. Then it's sufficient to provide the object code of the proprietary library as long as it can be ran/linked after making your changes to the published source in FOSS modules or the FOSS core and recompiling the whole thing.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems you are right, the SDKs for push notifications which I found (Google Firebase and Microsoft Azure) are under Apache license. So I changed the section to allow any OSI-approved open source license.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something that might be a problem is plugins and their dependencies can't be GPL unless they use the same GPL version as Lemmy, since Lemmy doesn't specify "or later" (LemmyNet/lemmy#5133). Unless plugins are somehow implemented in a way that GPL doesn't interpret as linking.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dullbananas The general conclusion by most others than me here seems to be that there are no such GPL limitations for plugins :/


# Future possibilities

Copy link

@bkil bkil Dec 16, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest we define the concept of plugin metadata from the start. Each plugin could be annotated by one or more of the following fields, inspired by apt (see also: other package managers):

  • an ID (using Java-like reverse DNS namespace notation)
  • a short name that fits a list
  • an icon
  • a short, few line description of what it does, possibly including indexed keywords (hashtags) if that's not present in a designated field
  • an easily parsable version number
  • copyright declaration (license SPDX or link, authors, date)
  • link to main landing page of own website
  • link to end user or sysadmin documentation (handbook, wiki, etc)
  • link to source code
  • link to a bug tracker, support contact channel or group, also how to report security issues
  • an exhaustive list of all API hooks the plugins would like permission to access during its operation (both to allow the admin to review it before installation and as defense in depth against bugs)
  • a list of plugins (possibly with minimum & maximum version number) which are required, suggested, conflicting and replaced by this one respectively

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a note about metadata near the beginning under File Structure. Though I want to keep it simple with only a few fields.

In the future we can add hooks for other user actions, as well as additional host functions.

We could also support implementing new post ranking algorithms in plugins.

Additionally it should be possible for plugins to define new API routes for additional functionality.

We may also consider to move some existing features into plugins. This applies to features which are mostly separate from core functionality, and which are only used by few instances. For example:
- Slur filter
- Custom emojis
- Taglines
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Theoretically it would be great if it was possible to externalize core code into plugins - both for separation of concerns, as well as to ensure the plugin structure is performant and complete. Practically it might be too much effort and for many things impossible

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once people start writing plugins, I could see it going in the opposite direction. A plugin becomes so popular that most instances use it, that we realize its an important enough feature that we should add it into lemmy core. That way everyone could use it without having to host a variety of plugins.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also makes sense, added.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any thoughts on how login plugins would work under this? For instance OIDC connectors (with overriding server usernames).

I admit I'm slightly biased here being on the FedID.me team and trying to bring federated and distributed credentials into existence ;)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It depends what those plugins need to do exactly. If they need extra database tables and api endpoints like LemmyNet/lemmy#4881 then it would be quite complicated. I definitely want to leave such things out of the initial implementation, and keep it simple for now.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, no API endpoints per-se, but likely something DB related to connect the local username to the OpenID identifier, similar to how Mastodon does it. (We just started experimenting with a FedID implementation with Mastodon via OIDC now.)

It's possible this information could be saved externally for Lemmy, so I'll start considering that.