Tutorial: Configure data stream retention

Serverless Stack

This tutorial demonstrates lifecycle retention, showing how to define, configure, and apply it. Keep in mind that the following options apply only to data streams that are managed by the data stream lifecycle.

1. What is data stream retention?
2. How to configure retention?
3. How is the effective retention calculated?
4. How is the effective retention applied?

You can verify if a data steam is managed by the data stream lifecycle via the get data stream lifecycle API:

				GET _data_stream/my-data-stream/_lifecycle
		

The result should look like this:

{
  "data_streams": [
    {
      "name": "my-data-stream",
      "lifecycle": {
        "enabled": true
      }
    }
  ]
}
		

1. The name of your data stream.
2. Ensure that the lifecycle is enabled, meaning this should be true.

We define retention as the least amount of time the data of a data stream are going to be kept in Elasticsearch. After this time period has passed, Elasticsearch is allowed to remove these data to free up space and/or manage costs.

We define 4 different types of retention:

• The data stream retention, or data_retention, which is the retention configured on the data stream level. It can be set via an index template for future data streams or via the PUT data stream lifecycle API for an existing data stream. When the data stream retention is not set, it implies that the data need to be kept forever.
• The global default retention, let’s call it default_retention, which is a retention configured via the cluster setting data_streams.lifecycle.retention.default and will be applied to all data streams managed by data stream lifecycle that do not have data_retention configured. Effectively, it ensures that there will be no data streams keeping their data forever. This can be set via the update cluster settings API.
• The global max retention, let’s call it max_retention, which is a retention configured via the cluster setting data_streams.lifecycle.retention.max and will be applied to all data streams managed by data stream lifecycle. Effectively, it ensures that there will be no data streams whose retention will exceed this time period. This can be set via the update cluster settings API.
• The effective retention, or effective_retention, which is the retention applied at a data stream on a given moment. Effective retention cannot be set, it is derived by taking into account all the configured retention listed above and is calculated as it is described here.

• Configure data retention at the data stream level using the data_retention setting. You can do this in two ways:

  • For a new data stream, the data_retention setting can be included in the index template that is applied when the data stream is created. You can use the create index template API, for example:

  				PUT _index_template/template
  					{
    "index_patterns": ["my-data-stream*"],
    "data_stream": { },
    "priority": 500,
    "template": {
      "lifecycle": {
        "data_retention": "7d"
      }
    },
    "_meta": {
      "description": "Template with data stream lifecycle"
    }
  }
  		

  • For an existing data stream, the data_retention setting can be configured using the PUT lifecycle API.

  				PUT _data_stream/my-data-stream/_lifecycle
  					{
    "data_retention": "30d"
  }
  		

  1. The retention period of this data stream is set to 30 days.
  Tip Serverless Stack GA 9.2.0

  To adjust the retention period of a data stream in Kibana, locate a data stream on the Streams page. A stream maps directly to a data stream. Next, select a stream to view its details and review the Retention tab to find out how it's managed before making your adjustments.

• By setting the global retention via the data_streams.lifecycle.retention.default and/or data_streams.lifecycle.retention.max that are applied on a cluster level. You can set these via the update cluster settings API. For example:

  				PUT /_cluster/settings
  					{
    "persistent" : {
      "data_streams.lifecycle.retention.default" : "7d",
      "data_streams.lifecycle.retention.max" : "90d"
    }
  }
  		

The effective is calculated in the following way:

• The effective_retention is the default_retention, when default_retention is defined and the data stream does not have data_retention.
• The effective_retention is the data_retention, when data_retention is defined and if max_retention is defined, it is less than the max_retention.
• The effective_retention is the max_retention, when max_retention is defined, and the data stream has either no data_retention or its data_retention is greater than the max_retention.

The above is demonstrated in the examples below:

Considering our example, if we retrieve the lifecycle of my-data-stream:

				GET _data_stream/my-data-stream/_lifecycle
		

We see that it will remain the same with what the user configured:

{
  "global_retention" : {
    "max_retention" : "90d",
    "default_retention" : "7d"
  },
  "data_streams": [
    {
      "name": "my-data-stream",
      "lifecycle": {
        "enabled": true,
        "data_retention": "30d",
        "effective_retention": "30d",
        "retention_determined_by": "data_stream_configuration"
      }
    }
  ]
}
		

1. The maximum retention configured in the cluster.
2. The default retention configured in the cluster.
3. The requested retention for this data stream.
4. The retention that is applied by the data stream lifecycle on this data stream.
5. The configuration that determined the effective retention. In this case it’s the data_configuration because it is less than the max_retention.

Retention is applied to the remaining backing indices of a data stream as the last step of a data stream lifecycle run. Data stream lifecycle will retrieve the backing indices whose generation_time is longer than the effective retention period and delete them. The generation_time is only applicable to rolled over backing indices and it is either the time since the backing index got rolled over, or the time optionally configured in the index.lifecycle.origination_date setting.