Use LD_PRELOAD for setting environment variables e.g. JAVA_TOOL_OPTIONS / OTEL_

[pavolloffay]

Component(s)

No response

Is your feature request related to a problem? Please describe.

At the moment the operator sets the env variables directly on the pod spec. The downside is if user defines the same variables in docker image or in config map. The operator does not know anything about those env vars. The solution might be to use LD_PRELOAD to set and override env vars.

The same approach is used by Lumigo operator and probably other vendors. https://github.com/lumigo-io/lumigo-kubernetes-operator/blob/88a3f78e5a32b965210659db11ef97f2a0c6f6c6/controller/src/mutation/mutate.go#L50

This would also resolve #1814

Describe the solution you'd like

Use LD_PRELOAD to set env vars for the instrumentation CR.

Describe alternatives you've considered

No response

Additional context

No response

[yuriolisa]

@pavolloffay, are you working on it or can I take this one?

[pavolloffay]

I haven't started. I think this needs maybe a POC and see what are the other implications.

[mmanciop]

OG of the Lumigo operator here :-)

We do this in the Dash0 operator (fully OSS), and while I personally built it as well in the Lumigo operator a couple of years ago, the injector there is not OSS iirc. We have used LD_PRELOAD technology at scale at Instana as well, and Dynatrace has been doing the same all along with their OneAgent. The implications mostly are three in my experience:

• The LD_PRELOAD object must be statically linked or you break anything that uses a different flavour of libc (yes, musl libc, I am looking at you; but older GLIBC versions are finicky as well IIRC)
• Unsupported Python versions, especially v2.7 and early v3, but they can be overcome with site-specific hooks
• Conflicting instrumentations built-in in the applications, but this is an issue that the current OpenTelemetry operator already faces

(Additional tidbits in this blog post.)

From Dash0's side, we'd be delighted to see LD_PRELOAD mechanics being adopted in the OpenTelemetry operator, and would love to help this happen.

[mmanciop]

Besides, the LD_PRELOAD technology is the same regardless of whether the injected runtime lives in a container or is running directly on the Linux host. If we develop this tech in the community, we would be able to provide, for example, DEB or RPM packages with all types of OTel automatic instrumentations that just work!

[pavolloffay]

@mmanciop thanks for looping in. We would appreciate collaboration on this feature 👍🏼

https://lumigo.io/blog/the-magic-behind-the-lumigo-kubernetes-operator/ mentions that the init-container is also used to copy the instrumentation/SDK. Does it mean that the docker image for the init container needs to bundle instrumentation for all languages?

[mmanciop]

@mmanciop thanks for looping in. We would appreciate collaboration on this feature 👍🏼

https://lumigo.io/blog/the-magic-behind-the-lumigo-kubernetes-operator/ mentions that the init-container is also used to copy the instrumentation/SDK. Does it mean that the docker image for the init container needs to bundle instrumentation for all languages?

Yes, the init container image must bundle all languages that the LD_PRELOAD can activate. And of course the LD_PRELOAD object itself.

[berezinsn]

Hi there,

I would like to inquire about any updates or progress on the feature related to concatenating JAVA_TOOL_OPTIONS with OpenTelemetry auto-instrumentation agent options.

As of now, it seems that it is only possible to concatenate environment variables explicitly defined within the deployment YAML under the env section. However, it appears that concatenation of the values from JAVA_TOOL_OPTIONS with the OpenTelemetry agent options (e.g., -javaagent or other configuration flags) is still not supported or automatic.

Could you please clarify whether this functionality is in the works or if there are any workarounds available? Any update would be greatly appreciated!

Thank you for your time and efforts.

[mmanciop]

Hi @berezinsn , it's a functionality we have working with the Dash0 operator and that I would love to contribute upstream. In your specific case, you maybe also use the _JAVA_OPTIONS environment variable, which is used by JVMs pretty much the same way JAVA_TOOL_OPTIONS is. Hope this helps in the meanwhile.

[mmanciop]

With respect to the PoC in #3731, we discussed in the SIG the two following points:

How can the LD_PRELOAD-based injector reuse configurations from other instrumentations, e.g., instrumentation image versions or Java Extensions?

Consensus was that the Instrumentation CR will change so that:

1. the Injector type will keep only the Image property, and instead, depending on the language to inject, reuse the Env and Resources setting from the specific language configurations.
2. The annotation to use the injector is to be renamed to inject-auto

How to deal with the languages (Python and .NET specifically) that cannot currently be injected reliably due to dependencies in the libc version used within the container, which cannot be determined at webhook time without user-provided configurations (which is a terminally bad UX imo). Also, the libc flavor can change between containers in the same pod, and therefore pod-based annotations are generally technically not good enough to provide full coverage to all containers.

This is an open topic. I reported about a promising PoC I am working on for a Zig-based injector, which would lift these limitations (e.g., by being able to determine the LibC flavor based on ELF metadata from /proc/self/exe), although it is something that is not ready to be tested.

[atoulme]

I am not sure using LD_PRELOAD here helps, because the code prepends to the env var our path to the jar ; that might mean we load multiple agents in the application, and it is my understanding that having multiple precompiler agents is not well supported by OpenTelemetry. So I'm not sure this is going to solve the premise of the issue.

[atoulme]

For host-based LD_PRELOAD instrumentation, we have a proposal up for a donation of the same functionality here: open-telemetry/community#2580
That approach is meant to specifically address system-wide, Linux-based instrumentation of applications. It works for multiple languages and is under review by the GC and TC.

[mmanciop]

I am not sure using LD_PRELOAD here helps, because the code prepends to the env var our path to the jar ; that might mean we load multiple agents in the application, and it is my understanding that having multiple precompiler agents is not well supported by OpenTelemetry. So I'm not sure this is going to solve the premise of the issue.

Technically, nothing prevents this injector to be modified to scrub from the env vars other agents it finds, or prevent adding an OTel one when other agents are found.

[atoulme]

OK, and which way are we leaning?

[mmanciop]

For host-based LD_PRELOAD instrumentation, we have a proposal up for a donation of the same functionality here: open-telemetry/community#2580 That approach is meant to specifically address system-wide, Linux-based instrumentation of applications. It works for multiple languages and is under review by the GC and TC.

If I read the code correctly, since that injector will link against a libc, it is viable when the packaging implies the LibC flavor (which is pretty much true for all Linux distros I know), but will likely crash processes that link against a different libc flavor (which is a commonplace risk in containers).

The existence of musl and Alpine do make injectors harder by several orders of magnitude…

[atoulme]

That's for sure. Does your LD_PRELOAD approach work with images built with scratch?

[mmanciop]

That's for sure. Does your LD_PRELOAD approach work with images built with scratch?

Yes, it’s entirely self-contained.

I am aware of an issue affecting processes that are dynamically linked, but that do not link against a libc (so they do not expose a __environ symbol), but that is the only issue it has had in about a year of Dash0 customer usage.