Add k8s annotation discovery blogpost

Co-authored-by: Dmitrii Anoshin <[email protected]>
Signed-off-by: ChrsMark <[email protected]>

Author: ChrsMark

content/en/blog/2025/otel-collector-k8s-discovery/index.md

@@ -0,0 +1,186 @@
+---
+title: Kubernetes annotation based discovery for OpenTelemetry Collector
+linkTitle: Kubernetes annotation discovery
+date: 2025-01-23
+author: >
+  [Dmitrii Anoshin](https://github.com/dmitryax) (Cisco/Splunk), [Christos
+  Markou](https://github.com/ChrsMark) (Elastic)
+sig: Collector
+issue: opentelemetry-collector-contrib#34427
+cSpell:ignore: Dmitrii Anoshin Markou
+---
+
+In the world of containers and [Kubernetes](https://kubernetes.io/),
+observability is crucial. Users need to know the status of their workloads at
+any given time. In other words, they need observability into moving objects.
+
+This is where the [OpenTelemetry Collector](/docs/collector) and its
+[receiver creator](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.117.0/receiver/receivercreator)
+component come in handy. Users can set up fairly complex monitoring scenarios
+with a self-service approach, following the principle of least privilege at the
+cluster level.
+
+The self-service approach is great, but how much self-service can it actually
+be? In this blog post, we will explore a newly added feature of the Collector
+that makes dynamic workload discovery even easier, providing a seamless
+experience for both administrators and users.
+
+## Automatic discovery for containers and pods
+
+Applications running on containers and pods become moving targets for the
+monitoring system. With automatic discovery, monitoring agents like the
+Collector can track changes at the container and pod levels and dynamically
+adjust the monitoring configuration.
+
+Today, the Collector—and specifically the receiver creator—can provide such an
+experience. Using the receiver creator, observability users can define
+configuration "templates" that rely on environment conditions. For example, as
+an observability engineer, I can configure my Collector to enable the
+[NGINX receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.117.0/receiver/nginxreceiver)
+when a NGINX pod is deployed on the cluster. The following configuration can
+achieve this:
+
+```yaml
+receivers:
+  receiver_creator:
+    watch_observers: [k8s_observer]
+    receivers:
+      nginx:
+        rule: type == "port" && port == 80 && pod.name matches "(?i)nginx"
+        config:
+          endpoint: 'http://`endpoint`/nginx_status'
+          collection_interval: '15s'
+```
+
+The above configuration will be enabled when a pod is discovered via the
+Kubernetes API that exposes port `80` (the known port for NGINX) and it's name
+matches the `nginx` keyword.
+
+This is great, and as an SRE or Platform Engineer managing an observability
+solution, you can rely on this to meet your users' needs for monitoring NGINX
+workloads. However, what happens if another team wants to monitor a different
+type of workload, such as Apache servers? They would need to inform your team,
+and you would need to update the configuration with a new conditional
+configuration block, take it through a pull request and review process, and
+finally deploy it. This deployment would require the Collector instances to
+restart for the new configuration to take effect. While this process might not
+be a big deal for some teams, there is definitely room for improvement.
+
+So, what if, as a Collector user, you could simply enable automatic discovery
+and then let your cluster users tell the Collector how their workloads should be
+monitored by annotating their pods properly? That sounds awesome, and it’s not
+actually something new. OpenTelemetry already supports auto-instrumentation
+through the Operator
+([documentation](https://opentelemetry.io/docs/kubernetes/operator/automatic/)),
+allowing users to instrument their applications automatically just by annotating
+their pods. In addition, this is a feature that other monitoring agents in the
+observability industry already support, and users are familiar with it.
+
+All this motivation led the OpenTelemetry community
+([GitHub issue](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/17418))
+to create a similar feature for the Collector. We are happy to share that
+autodiscovery based on Kubernetes annotations is now supported in the Collector
+([GitHub issue](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/34427))!
+
+## The solution
+
+The solution is built on top of the existing functionality provided by the
+[Kubernetes observer](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.117.0/extension/observer/k8sobserver)
+and
+[receiver creator](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.117.0/receiver/receivercreator).
+
+The K8s observer notifies the receiver creator about the objects appearing in
+the K8s cluster and provides all the information about them. In addition to the
+K8s object metadata, the observer supplies information about the discovered
+endpoints that the collector can connect to. This means that each discovered
+endpoint can potentially be used by a particular scraping receiver to fetch
+metrics data.
+
+Each scraping receiver has a default configuration with only one required field:
+`endpoint`. Given that the endpoint information is provided by the Kubernetes
+observer, the only information that the user needs to provide explicitly is
+which receiver/scraper should be used to scrape data from a discovered endpoint.
+That information can be configured on the collector, but as mentioned before,
+this is inconvenient. A much more convenient place to define which receiver can
+be used to scrape telemetry from a particular pod is the pod itself. Pod’s
+annotations is the natural place to put that kind of detail. Given that the
+receiver creator has access to the annotations, it can instantiate the proper
+receiver with the receiver’s default configuration and discovered endpoint.
+
+The following annotation instructs the receiver creator that this particular pod
+runs NGINX, and the
+[NGINX receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.117.0/receiver/nginxreceiver)
+can be used to scrape metrics from it:
+
+```yaml
+io.opentelemetry.discovery.metrics/scraper: nginx
+```
+
+Apart from that, the discovery on the pod need to be explicitly enabled with the
+following annotation:
+
+```yaml
+io.opentelemetry.discovery.metrics/enabled: 'true'
+```
+
+In some scenarios, the default receiver’s configuration is not suitable for
+connecting to a particular pod. In that case, it’s possible to define custom
+configuration as part of another annotation:
+
+```yaml
+io.opentelemetry.discovery.metrics/config: |
+  endpoint: "http://`endpoint`/nginx_status"
+  collection_interval: '20s'
+  initial_delay: '20s'
+  read_buffer_size: '10'
+```
+
+It’s important to mention that the configuration defined in the annotations
+cannot point the receiver creator to another pod. The collector will reject such
+configurations.
+
+In addition to the metrics scraping, the annotation-based discovery also
+supports log collection with filelog receiver. The following annotation can be
+used to enable log collection on a particular pod:
+
+```yaml
+io.opentelemetry.discovery.logs/enabled: 'true'
+```
+
+Similar to metrics, an optional configuration can be provided in the following
+form:
+
+```yaml
+io.opentelemetry.discovery.logs/config: |
+  max_log_size: "2MiB"
+  operators:
+  - type: container
+    id: container-parser
+  - type: regex_parser
+    regex: '^(?P<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P<sev>[A-Z]*) (?P<msg>.*)$'
+```
+
+If the set of filelog receiver operators needs to be changed, the full list,
+including the default container parser, has to be redefined because list config
+fields are entirely replaced when merged into the default configuration struct.
+
+The discovery functionality has to be explicitly enabled in the receiver creator
+just by adding the following configuration field:
+
+```yaml
+receivers:
+  receiver_creator:
+    watch_observers: [k8s_observer]
+    discovery:
+      enabled: true
+```
+
+## Conclusion - Wrapping up
+
+If you are an OpenTelemetry Collector user on Kubernetes and you find this new
+feature interesting, go ahead and visit the official
+[documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.117.0/receiver/receivercreator/README.md#generate-receiver-configurations-from-provided-hints)
+to learn more! And if you give it a try let us know what you think. Don't
+hesitate to reach out to us in the official CNCF
+[Slack workspace](https://slack.cncf.io/) and specifically the `#otel-collector`
+channel.