Refactor FeatureFlags

[finnegancarroll]

Description

There are a handful of competing ways to set feature flags today.

1. FeatureFlags.initializeFeatureFlags
   Takes a setting object and overwrites FeatureFlags.settings. Node invokes this to set flags on startup, but it is also used throughout unit tests to dynamically update and clear feature flags. Provides no way to set a single flag without rewriting all settings.

2. System.setProperty:
   Sets the feature flag as a JVM property. Calls to FeatureFlags.isEnabled() will check System.getProperty() and see this flag. Allows dynamic setting of individual feature flags.

3. FeatureFlagSetter.set:
   A small wrapper around the above System.setProperty() which additionally tracks which flags it sets. Used only in tests. Tracks set properties and exposes clearAll() to unset all tracked properties which is invoked during OpenSearchTestCase teardown to ensure a "clean slate" for each test.

Together these implementations provide the desired functionality for feature flags across test and non-test use cases but present some issues:

• Non of the implementations are thread safe as feature flags are global/static and may be modified at any time.
• System properties persist for the lifetime of the JVM and can carry over to other tests run on the same JVM, polluting other test environments if not cleared.
• System.getProperty() can be slow and have performance implications when feature flags are retrieved in a hot path.
• When not using System.setProperty(), clearing or setting any single feature flag resets all flags.

This PR refactors FeatureFlags to consolidate feature flag functionality and remove usage of JVM system properties in all cases except initializeFeatureFlags which is called once by Node on startup.

Specific changes include:

• Replaces FeatureFlags immutable internal settings with a map.
• initializeFeatureFlags now reads and sets feature flags from JVM system properties.
• isEnabled no longer checks JVM system properties to mitigate performance impact.
• For test use cases functionality for setting/unsetting flags is provided behind FeatureFlags.TestUtils for clarity.
  • @LockFeatureFlag annotation provided as a thread safe way to set a feature flag for the duration of the test case.
  • FeatureFlags.TestUtils.with for thread safe execution of a critical section with provided feature flag enabled.
  • FeatureFlags.TestUtils.FlagLock for directly accessing a feature flag's lock.

Related Issues

Resolves #16519

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for fdec16b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for f1d5490: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 1228bda: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 4a3e60f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[finnegancarroll]

Verifying isEnabled is represented less in big5 term flame graph per #16519.

Feature branch:

Main branch:

[github-actions]

❌ Gradle check result for 4bc33cd: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 94fd7af: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for e8ad6be: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[finnegancarroll]

{"run-benchmark-test": "id_5"}

[github-actions]

The Jenkins job url is https://build.ci.opensearch.org/job/benchmark-pull-request/2624/ . Final results will be published once the job is completed.

[finnegancarroll]

{"run-benchmark-test": "id_3"}

[github-actions]

✅ Gradle check result for e8ad6be: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 95.12195% with 4 lines in your changes missing coverage. Please review.

Project coverage is 72.37%. Comparing base (1166998) to head (e8ad6be).
Report is 12 commits behind head on main.

Files with missing lines	Patch %	Lines
.../java/org/opensearch/common/util/FeatureFlags.java	95.12%	4 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #17611      +/-   ##
============================================
- Coverage     72.48%   72.37%   -0.11%     
+ Complexity    65771    65694      -77     
============================================
  Files          5311     5311              
  Lines        304973   305063      +90     
  Branches      44229    44241      +12     
============================================
- Hits         221045   220799     -246     
- Misses        65830    66221     +391     
+ Partials      18098    18043      -55     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[finnegancarroll]

distance_amount_agg is just barely showing a ~5% regression in 90+ percentile latency.
#17611 (comment)