[META] Productionalizing Client/Server GRPC in OpenSearch

[amberzsy]

Please describe the end goal of this project

Please comment on this issue with any feedback on our recent experimental release.
This is a meta issue for all work relating to Client to Server GRPC.

Supporting References

• [RFC] Protobuf in OpenSearch
• [META] Leverage protobuf for serializing select node-to-node objects
• [META] Protobuf for Search API
• gRPC support for client Search API (initial client<>server benchmark result for Search api)

Issues

Core Work

GRPC Server Core

• [META] gRPC Server #16556
• [Feature Request] Security plugin integration for grpc-transport plugin  #16905

Search APIs

• [RFC] gRPC-based API for Search  #15190 (proposal)
• [Feature] GRPC Search APIs #16783

Document APIs

• [Feature] GRPC Document APIs #16784

Index APIs

• [Feature Request] GRPC Index APIs #16785

Other APIs

• [Feature] GRPC Cluster APIs
• [Feature] GRPC Nodes APIs
• [Feature] GRPC CAT APIs
• [Feature] GRPC Ingest APIs

TODO: Other APIs from the public doc are omitted for brevity, and can be added later as we become closer to working on them.

Proto conversion

Protobuf support

• [PROPOSAL] Publish protobuf files. opensearch-api-specification#532
• [PROPOSAL] Spec to Protobuf Conversion Rules and Tooling opensearch-api-specification#677
• [FEATURE] Spec Conversion to Protobufs opensearch-api-specification#707

CI/CD

• [PROPOSAL] Core <> Spec/Protobuf CI/CD opensearch-api-specification#653
• [Feature] CI/CD for Protobufs

Benchmarking

• GRPC Search API benchmark #16710
• GRPC Document API benchmark #16711

Related

• [META] GRPC Ecosystem Integration #16786

Milestone goal

GRPC - core repo

Protobuf & CI/CD

Related component

Other

[finnegancarroll]

Noting some additional work streams for core and OSB support:

• Extensibility support for plugins
• OSB gRPC client support

[finnegancarroll]

@amberzsy thanks for adding these milestones!
Just want to note that after the 3.0.0-beta1 release there will be a soft feature freeze before 3.0 GA. Any larger additions to core might want to be prioritized for beta release since they will be under greater scrutiny after. Initial Bulk/Search API/Security plugin integration likely fall into this category where as tooling, performance, and plugin related changes probably have a lot more leeway.

[amberzsy]

@amberzsy thanks for adding these milestones! Just want to note that after the 3.0.0-beta1 release there will be a soft feature freeze before 3.0 GA. Any larger additions to core might want to be prioritized for beta release since they will be under greater scrutiny after. Initial Bulk/Search API/Security plugin integration likely fall into this category where as tooling, performance, and plugin related changes probably have a lot more leeway.

yeah, good point. few questions:

1. will security plugin integration be part of 3.0 release? does it have dependency on bulk/search api implementation? (i assume it's not)
2. When's the beta release? or more specific what would be potential process and timeline look like for 3.0 release?
3. for opensearch-api-spec, is it the same release process or separate? i guess it's not the same one.
4. as the core bulk/search api would depend on new opensearch-protobufs repo, will it be any code freeze on new repo etc?

[finnegancarroll]

1. will security plugin integration be part of 3.0 release? does it have dependency on bulk/search api implementation? (i assume it's not)

The security plugin changes won't have any impact on bulk/search api. Just calling it out as well since it is planned for 3.0 and preferably makes it into beta release. PR is in review here with security plugin initial changes here.

3. When's the beta release? or more specific what would be potential process and timeline look like for 3.0 release?

Release timeline can be found here with release candidate generation to start 4/1.

4. for opensearch-api-spec, is it the same release process or separate? i guess it's not the same one.

I think the opensearch-api-spec itself doesn't do releases. However client projects periodically pull the latest main branch and generate code for their releases which mirror core.

5. as the core bulk/search api would depend on new opensearch-protobufs repo, will it be any code freeze on new repo etc?

I imagine only large changes to the compiled protobuf libraries consumed by core would be impacted. Tooling/CI/client generation changes are probably not a concern. @peterzhuamazon will know better.