Kubernetes Environments

Elastic Stack Serverless Observability

The recommended OTel architecture for Kubernetes clusters includes a set of OpenTelemetry collectors in different forms. The following diagram shows the different forms:

The Collector in Daemon form is deployed on each Kubernetes node to collect nodes-local logs and host metrics.

The daemon collector also receives telemetry from applications instrumented with OTel SDKs and running on corresponding nodes.

That Collector enriches the application telemetry with resource information such as host and Kubernetes metadata.

All data is then being sent through OTLP to the OTel or EDOT Gateway Collector.

The Collector in Cluster form collects Kubernetes cluster-level metrics and sends them to the OTel or EDOT Gateway Collector using OTLP.

The OTel or EDOT Collector in Gateway form gathers the OTel data from all other collectors and ingests it into the Elastic backend.

For self-managed and Elastic Cloud Hosted deployment models the Gateway Collector does some additional pre-processing of data.

See the recommended architectures per Elastic deployment scenarios:

Elastic Cloud Serverless provides a Managed OTLP Endpoint for ingestion of OpenTelemetry data.

For a Kubernetes setup, that means the Gateway Collector passes through the OTel data in native format using the OTLP protocol to the managed OTLP endpoint. There is no need for the Gateway Collector to do any Elastic-specific pre-processing.

With Elastic Cloud Hosted (ECH), OTel data is being directly ingested into the Elastic-hosted Elasticsearch instance.

The Gateway Collector needs to do some preprocessing, aggregation of metrics and, finally, it uses the Elasticsearch exporter to ingest data into ECH.

While the Daemon and Cluster collectors, as well as the OTel SDKs, can stay fully vendor agnostic or upstream, the Gateway Collector needs to be either an EDOT Collector or a custom, EDOT-like Collector containing the required components and pre-processing pipelines.

If required, users can build their custom, EDOT-like Collector following these instructions.

With a self-managed scenario the Gateway Collector ingests data directly into the self-managed Elasticsearch instance.

The Gateway Collector does some preprocessing and aggregation of OTel data before ingesting it into Elasticsearch.

While the Daemon and Cluster collectors, as well as the OTel SDKs, can stay fully vendor agnostic or upstream, the Gateway Collector needs to be either an EDOT Collector or a custom, EDOT-like Collector containing the required components and pre-processing pipelines.