[Pull-based Ingestion] Add Kinesis plugin support

[yupeng9]

Description

This PR adds support Kinesis as a plugin for pull-based ingestion. Because Kinesis has some differences from Kafka of modeling the stream, this requires some changes to the core stream poller. For example, Kinesis generates a sequence number for a new record, so it means it's not always possible to get the next sequence number when the offset is already at the head of the stream. Moreover, the sequence numbers between adjacent records are not consecutive, though they are still incremental. To accommodate Kinesis's behavior, we need to change the stream poller to track the last successful poll instead of getting the next pointer, which may not exist in Kinesis if the last offset is at the head of the stream.

Related Issues

Part of this META issue

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for f267858: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for a09ead0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for a09ead0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 3031cf8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 74e7329:

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for d3735a8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 83ab1f8:

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[msfroh]

This is looking pretty good!

Just left a couple of minor comments and some "bookkeeping" comments to capture Slack discussion results in GitHub.

[github-actions]

❌ Gradle check result for 83ab1f8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 15c571e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 2a99aa4: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 72.72727% with 42 lines in your changes missing coverage. Please review.

Project coverage is 72.50%. Comparing base (47d3655) to head (2a99aa4).
Report is 11 commits behind head on main.

Files with missing lines	Patch %	Lines
...pensearch/plugin/kinesis/KinesisShardConsumer.java	72.00%	16 Missing and 5 partials ⚠️
.../org/opensearch/plugin/kinesis/SequenceNumber.java	60.86%	5 Missing and 4 partials ⚠️
...pensearch/plugin/kafka/KafkaPartitionConsumer.java	53.84%	4 Missing and 2 partials ⚠️
...a/org/opensearch/plugin/kinesis/KinesisPlugin.java	0.00%	3 Missing ⚠️
...nsearch/plugin/kinesis/KinesisConsumerFactory.java	71.42%	2 Missing ⚠️
...a/org/opensearch/index/engine/IngestionEngine.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #17615      +/-   ##
============================================
- Coverage     72.53%   72.50%   -0.04%     
- Complexity    65826    65832       +6     
============================================
  Files          5311     5317       +6     
  Lines        305073   305211     +138     
  Branches      44243    44258      +15     
============================================
- Hits         221293   221287       -6     
- Misses        65688    65843     +155     
+ Partials      18092    18081      -11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.