Customize built-in policies

Stack

Elasticsearch includes a set of built-in ILM policies that govern how managed indices transition as they age. This guide demonstrates how you can customize the lifecycle of a managed index, to adjust how the index transitions across data tiers and what actions, such as downsampling or shrinking, are performed on the index during each lifecycle phase.

Setting a custom ILM policy is useful when you have a specific set of indices, for example a set of Kubernetes logs which can grow to be quite large in volume, for which you don't want to use the default data retention duration and other ILM settings.

Elastic Agent uses the following set of built-in ILM policies to manage backing indices for its data streams:

logs@lifecycle
logs-otel@lifecycle
metrics@lifecycle
metrics-otel@lifecycle
synthetics@lifecycle
traces@lifecycle
traces-otel@lifecycle

This tutorial covers customizing the way ingested logging data is managed. Rather than use the default lifecycle settings from the built-in logs@lifecycle ILM policy, you can use the Index Lifecycle Policies feature in Kibana to tailor a new policy based on your application’s specific performance, resilience, and retention requirements. This involves three main steps:

Create a duplicate of the logs@lifecycle policy.
Modify the new policy to suit your requirements.
Apply the new policy to your log data using a logs@custom component template.

Tip

If you're using Elastic integrations and are not yet familiar with which data streams are associated with them, refer to Manage the lifecycle policy for integrations data.
If you're looking for a more advanced use case, such as customizing an ILM policy for a selected set of data streams in one or more integrations or namespaces, check the set of tutorials in Customize data retention policies in the Fleet and Elastic Agent reference documentation.

These tutorials go into greater depth about creating and using @custom component templates. For example, the tutorial Apply an ILM policy to all data streams generated from Fleet integrations across all namespaces shows how to create and use the logs@custom and metrics@custom component templates to customize ILM policies associated with data streams in integrations.

Scenario

You want to send log files to an Elasticsearch cluster so you can visualize and analyze the data. This data has the following retention requirements:

When the primary shard size of the write index reaches 50GB or the index is 30 days old, roll over to a new index.
After rollover, keep indices in the hot data tier for 30 days.
30 days after rollover:
- Move indices to the warm data tier.
- Set replica shards to 1.
- Force merge multiple index segments to free up the space used by deleted documents.
Delete indices 90 days after rollover.

Prerequisites

To complete this tutorial, you’ll need:

An Elasticsearch cluster with hot and warm data tiers.
- Elastic Cloud Hosted: Elastic Stack deployments on Elastic Cloud include a hot tier by default. To add a warm tier, edit your deployment and click Add capacity for the warm data tier.
  
  ×
- Self-managed cluster: Assign data_hot and data_warm roles to nodes as described in Data tiers.
  
  For example, include the data_warm node role in the elasticsearch.yml file of each node in the warm tier:
```
node.roles: [ data_warm ]
		
```
A host with Elastic Agent installed and configured to send logs to your Elasticsearch cluster.

Duplicate the policy

Elastic Agent uses data streams with an index pattern of logs-*-* to store log monitoring data. The managed logs@lifecycle ILM policy automatically manages backing indices for these data streams.

If you don’t want to use the policy defaults, then you can customize the managed policy and then save it as a new policy. You can then use the new policy in related component templates and index templates.

Warning

You should never edit managed policies directly. Changes to managed policies might be rolled back or overwritten.

To save the logs@lifecycle policy as a new policy in Kibana:

Go to the Index Lifecycle Policies management page using the navigation menu or the global search field.
Toggle Include managed system policies.
Select the logs@lifecycle policy.
On the Edit policy logs page, toggle Save as new policy, and then provide a new name for the policy, for example, logs-custom.

The logs@lifecycle policy uses the recommended rollover defaults: Start writing to a new index when the primary shard size of the current write index reaches 50GB or the index becomes 30 days old.

To view or change the rollover settings, click Advanced settings for the hot phase. Then disable Use recommended defaults to display the rollover settings.

Modify the policy

The default logs@lifecycle policy is designed to prevent the creation of many tiny daily indices. You can modify your copy of the policy to meet your performance requirements and manage resource usage.

Activate the warm phase and click Advanced settings.
1. Set Move data into phase when to 30 days old. This moves indices to the warm tier 30 days after rollover.
2. Enable Set replicas and change Number of replicas to 1.
3. Enable Force merge data and set Number of segments to 1.
×
In the warm phase, click the trash icon to enable the delete phase.

×

In the delete phase, set Move data into phase when to 90 days old. This deletes indices 90 days after rollover.

×
Click Save as new policy.

Tip

Copies of managed ILM policies are also marked as Managed. You can use the Create or update lifecycle policy API to update the _meta.managed parameter to false.

Apply the policy

To apply your new ILM policy to the logs index template, create or edit the logs@custom component template.

Using @custom component templates

A @custom component template allows you to customize the mappings and settings of the managed index templates, without having to override them or their main component templates.

Many Elasticsearch managed index templates include one or more @custom component templates. A @custom component template must first be created before it can be used, and its name must exactly match the name specified in the managed index template in order to be applied automatically to indices as they're created.

For example, if you're ingesting OpenTelemetry (OTel) logs, any OTel log data streams and their backing indices are configured by the logs-otel@template managed index template. That index template automatically applies settings defined in the logs@custom and the logs-otel@custom template when they exist.

Go to Index Management > Index Templates and select any managed index to view the @custom component templates associated with it.

A screenshot showing the logs@custom and logs-otel@custom component templates associated with the logs-otel@template index template.

Tip

If you want your ILM changes to apply only to specific indices, you can create a custom index template directly instead of modifying the custom component template. Use the Index management page in Kibana or the index template API to create a new template.

Click on the Component Template tab and click Create component template.
Under Logistics, name the component template logs@custom.

Under Index settings, set the ILM policy name created in the previous step:

		{
    "index": {
        "lifecycle": {
            "name": "logs-custom"
        }
    }
}
		
	

Continue to Review, and then click Save component template.
Click the Index Templates, tab, and then select the logs index template.
In the summary, view the Component templates list. logs@custom should be listed.