[BUG] OpenSearch security plugin unable to load after a pod is killed due to opensearch.yaml configuration.

[JonasErstad]

What is the bug?
When a pod running OpenSearch is terminated by the operating system and restarted, the new pod is unable to load the OpenSearch Security plugin. We've found that this happens because OpenSearch uses the presence of plugins.security keys in the opensearch.yaml configuration file to determine whether the security plugin is configured. Which is still present after the process is killed.
When we delete the security.plugins entries in opensearch.yaml the pod is able to start up again with no errors.

In the logs we can see that opensearch.yaml is configured for Security.
Detected OpenSearch Version: 2.18.0 Detected OpenSearch Security Version: 2.18.0.0 /usr/share/opensearch/config/opensearch.yml seems to be already configured for Security. Quit.

And in the stacktraces we can see errors when loading the plugin:

org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.18.0.jar:2.18.0]
        ...

Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:805) ~[opensearch-2.18.0.jar:2.18.0]
	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:744) ~[opensearch-2.18.0.jar:2.18.0]
        ...

Caused by: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74) ~[?:?]
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
        ...

Caused by: org.opensearch.OpenSearchException: Unable to read the file root-ca.pem. Please make sure this files exists and is readable regarding to permissions
	at org.opensearch.security.ssl.config.SslCertificatesLoader.resolvePath(SslCertificatesLoader.java:165) ~[?:?]
	at org.opensearch.security.ssl.config.SslCertificatesLoader.loadConfiguration(SslCertificatesLoader.java:85) ~[?:?]
	at org.opensearch.security.ssl.SslSettingsManager.loadConfigurations(SslSettingsManager.java:137) ~[?:?]
        ...

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Kill the process on the worker-node that is running the pod.

What is the expected behavior?
A pod should be able to load the security plugin even if its killed by the operating system.

What is your host/environment?
OpenSearch Version: 2.18.0

Do you have any additional context?
We are running a rke2 kubernetes cluster on ubuntu 24.04.
The issue was also present for us on version 2.15.0 of OpenSearch.

[cwperks]

[Triage] @JonasErstad Thank you for filing this issue. When running the install_demo_configuration.sh script, if it detects that security is already configured it aborts and does not proceed to install the demo configuration. Have you customized security with your own certs and settings?

[JonasErstad]

No we have not customized security with our own certs and settings.

The issue for us is that it does indeed skip the installation of the demo configuration. When the container that initially installed the demo configuration is killed and restarted, the plugins are not reinstalled, as per the configuration set in opensearch.yml. Which leads to the failure of loading the plugin class seen in the stacktraces.

Currently, our workaround is to use a postStart lifecycle hook that truncates the opensearch.yml file upon startup.

[Nagpraveenbv]

Im getting a vm maxmapcount issues after truncating opensearch yaml @JonasErstad

[DarshitChanpura]

@JonasErstad Would you mind opening a forum discussion at https://forum.opensearch.org.