Securing the metrics endpoint

ECK

If you're using ECK 2.16 or earlier, then after you enable your metrics endpoint, you need to secure it. To enable RBAC and TLS on the metrics endpoint, follow the instructions in the following sections depending on whether you installed ECK through the Helm chart or the manifests.

The ECK operator metrics endpoint is secured by default beginning in version 3.0.0. If you're using ECK 3.0.0 or later, then you might want to use the information on this page to provide your own TLS certificate.

Using the operator Helm chart

If you installed ECK through the Helm chart commands listed in Install using a Helm chart, you can set config.metrics.secureMode.enabled to true and both RBAC and TLS/HTTPs will be enabled for the metrics endpoint.

Using your own TLS certificate for the metrics endpoint when using the Helm chart

By default, a self-signed certificate will be generated for use by the metrics endpoint. If you want to use your own TLS certificate for the metrics endpoint, you can provide the config.metrics.secureMode.tls.certificateSecret to the Helm chart. The certificateSecret should be the name of an existing Kubernetes Secret that contains both the TLS certificate and the TLS private key. The following keys are supported within the secret:

tls.crt: The PEM-encoded TLS certificate
tls.key: The PEM-encoded TLS private key

The easiest way to create this secret is to use the kubectl create secret tls command. For example:

		kubectl create secret tls eck-metrics-tls-certificate -n elastic-system --cert=/path/to/tls.crt --key=/path/to/tls.key
		
	

Providing this secret is sufficient to use your own certificate if it is from a trusted Certificate Authority. If the certificate is not signed by a trusted CA and you are using Prometheus to scrape the metrics, you have the following options:

Disable TLS verification.
- Set serviceMonitor.insecureSkipVerify to true to disable TLS validation in the ServiceMonitor generated by the eck-operator Helm chart.
Provide the Certificate Authority to Prometheus.
- Set serviceMonitor.insecureSkipVerify to false to enable TLS validation.
- Set serviceMonitor.caSecret to the name of an existing Kubernetes secret within the Prometheus namespace that contains the CA in PEM format in a file called ca.crt.
- Set the spec.secrets field of the Prometheus custom resource, or prometheus.prometheusSpec.secrets when using the Helm chart such that the CA secret is mounted into the Prometheus pod at serviceMonitor.caMountDirectory (assuming you are using the Prometheus operator). See the ECK Helm chart values file for more information.

Refer to Prometheus requirements for more information on creating the CA secret.

Using the operator manifests

If you installed ECK through using the manifests using the commands listed in Install ECK using the YAML manifests, some additional changes are required to enable secure metrics.

Enable the metrics port in the ConfigMap and set the metrics-secure setting to true.

		cat <<EOF | kubectl apply -f -
kind: ConfigMap
apiVersion: v1
metadata:
  name: elastic-operator
  namespace: elastic-system
data:
  eck.yaml: |-
    log-verbosity: 0
    metrics-port: 8081
    metrics-host: 0.0.0.0
    metrics-secure: true
    container-registry: docker.elastic.co
    max-concurrent-reconciles: 3
    ca-cert-validity: 8760h
    ca-cert-rotate-before: 24h
    cert-validity: 8760h
    cert-rotate-before: 24h
    disable-config-watch: false
    exposed-node-labels: [topology.kubernetes.io/.*,failure-domain.beta.kubernetes.io/.*]
    set-default-security-context: auto-detect
    kube-client-timeout: 60s
    elasticsearch-client-timeout: 180s
    disable-telemetry: false
    distribution-channel: all-in-one
    validate-storage-class: true
    enable-webhook: true
    webhook-name: elastic-webhook.k8s.elastic.co
    webhook-port: 9443
    operator-namespace: elastic-system
    enable-leader-election: true
    elasticsearch-observation-interval: 10s
    ubi-only: false
EOF
		
	

Add an additional ClusterRole and ClusterRoleBinding for the ECK operator.

		cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: elastic-operator-metrics-auth-role
rules:
- apiGroups:
  - authentication.k8s.io
  resources:
  - tokenreviews
  verbs:
  - create
- apiGroups:
  - authorization.k8s.io
  resources:
  - subjectaccessreviews
  verbs:
  - create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: elastic-operator-metrics-auth-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: elastic-operator-metrics-auth-role
subjects:
- kind: ServiceAccount
  name: elastic-operator
  namespace: elastic-system
EOF
		
	

Add a Service to expose the metrics endpoint.

		cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  labels:
    control-plane: elastic-operator
    app.kubernetes.io/component: metrics
  name: elastic-operator-metrics
  namespace: elastic-system
spec:
  ports:
  - name: https
    port: 8080
    protocol: TCP
    targetPort: metrics
  selector:
    control-plane: elastic-operator
EOF
		
	

If you're using the Prometheus operator, add a ServiceMonitor to allow Prometheus to scrape the metrics endpoint.

		cat <<EOF | kubectl apply -f -
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: elastic-operator
  namespace: elastic-system
spec:
  namespaceSelector:
    matchNames:
      - elastic-system
  selector:
    matchLabels:
      control-plane: elastic-operator
      app.kubernetes.io/component: metrics
  endpoints:
  - port: https
    path: /metrics
    scheme: https
    interval: 30s
    tlsConfig:
      insecureSkipVerify: true
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
EOF
		
	

Using your own TLS certificate for the metrics endpoint when using the manifests

By default a self-signed certificate will be generated for use by the metrics endpoint. If you want to use your own TLS certificate for the metrics endpoint you will need to follow the previous instructions to enable secure metrics as well as the following steps:

Create a Secret containing the TLS certificate and TLS private key. The following keys are supported within the secret:
- tls.crt - The PEM-encoded TLS certificate
- tls.key - The PEM-encoded TLS private key
The easiest way to create this secret is to use the kubectl create secret tls command. For example:
```
kubectl create secret tls my-tls-secret -n elastic-system --cert=/path/to/tls.crt --key=/path/to/tls.key
		
```

Patch the StatefulSet to include the tls.crt and tls.key as a volume and mount it into the manager container.

		kubectl patch sts -n elastic-system elastic-operator --patch-file=/dev/stdin <<-EOF
spec:
  template:
    spec:
      containers:
        - name: manager
          volumeMounts:
          - mountPath: "/tmp/k8s-metrics-server/serving-certs"
            name: tls-certificate
            readOnly: true
      volumes:
      - name: conf
        configMap:
          name: elastic-operator
      - name: cert
        secret:
          defaultMode: 420
          secretName: elastic-webhook-server-cert
      - name: tls-certificate
        secret:
          defaultMode: 420
          secretName: eck-metrics-tls-certificate
EOF
		
	

If you're mounting the TLS secret to a different directory, the metrics-cert-dir setting in the operator configuration has to be adjusted accordingly.

If required, patch the ServiceMonitor. This is required if you are adjusting the insecureSkipVerify field to false.

		kubectl patch servicemonitor -n elastic-system elastic-operator --patch-file=/dev/stdin <<-EOF
spec:
  endpoints:
  - port: https
    path: /metrics
    scheme: https
    interval: 30s
    tlsConfig:
      insecureSkipVerify: false
      caFile: /etc/prometheus/secrets/{secret-name}/ca.crt
      serverName: elastic-operator-metrics.elastic-system.svc
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
EOF
		
	

See Prometheus requirements for more information on creating the CA secret.