Set up a time series data stream

Create an index lifecycle policy (optional)

Serverless Unavailable Stack

In most cases, you can use a data stream lifecycle to manage your time series data stream. If you're using data tiers in Elastic Stack, you can use index lifecycle management.

				PUT _ilm/policy/my-weather-sensor-lifecycle-policy
					{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_primary_shard_size": "50gb"
          }
        }
      }
      // Additional phases (warm, cold, delete) as needed
      }
    }
  }
}
		

Create an index template

The structure of a time series data stream is defined by an index template. Create an index template with the following required elements and settings:

• Index patterns: One or more wildcard patterns matching the name of your TSDS, such as weather-sensors-*. For best results, use the data stream naming scheme.
• Data stream object: The template must include "data_stream": {}.
• Time series mode: Set index.mode: time_series.
• Field mappings: Define at least one dimension field and typically one or more metric fields:
  • Dimensions: To define a dimension, set time_series_dimension to true. For details, refer to Dimensions.
    • To define dimensions dynamically, you can use a pass-through object. For details, refer to Defining sub-fields as time series dimensions.
  • Metrics: To define a metric, use the time_series_metric mapping parameter. For details, refer to Metrics.
  • Timestamp (optional): Define a date or date_nanos mapping for the @timestamp field. If you don't specify a mapping, Elasticsearch maps @timestamp as a date field with default options.
  • Serverless Unavailable Stack Lifecycle management: For Elastic Stack, include lifecycle settings to enable automatic rollover and prevent indices from growing too large.
    • Set "lifecycle": { "enabled": true }.
    • If you created an ILM policy in step 1, reference it with index.lifecycle.name.
  • Other settings (optional): Additional index settings, such as index.number_of_replicas, for the data stream's backing indices.
• Priority: Set the priority higher than 200 to avoid collisions with built-in templates.

Example index template PUT request:

				PUT _index_template/my-weather-sensor-index-template
					{
  "index_patterns": ["metrics-weather_sensors-*"],
  "data_stream": { },
  "template": {
    "settings": {
      "index.mode": "time_series",
      "index.lifecycle.name": "my-lifecycle-policy",
      "lifecycle": { 
        "enabled": true
      }
    },
    "mappings": {
      "properties": {
        "sensor_id": {
          "type": "keyword",
          "time_series_dimension": true
        },
        "location": {
          "type": "keyword",
          "time_series_dimension": true
        },
        "temperature": {
          "type": "half_float",
          "time_series_metric": "gauge"
        },
        "humidity": {
          "type": "half_float",
          "time_series_metric": "gauge"
        },
        "@timestamp": {
          "type": "date"
        }
      }
    }
  },
  "priority": 500,
  "_meta": {
    "description": "Template for my weather sensor data"
  }
}
		

1. Elastic Stack only
2. Elastic Stack only

Create the time series data stream and add data

After creating the index template, you can create a time series data stream by indexing a document. The TSDS is created automatically when you index the first document, as long as the index name matches the index template pattern. You can use a bulk API request or a POST request.

				PUT metrics-weather-sensors/_bulk
					{ "create":{ } }
{ "@timestamp": "2099-05-06T16:21:15.000Z", "sensor_id": "SENSOR-001", "location": "warehouse-A", "temperature": 26.7,"humidity": 49.9 }
{ "create":{ } }
{ "@timestamp": "2099-05-06T16:25:42.000Z", "sensor_id": "SENSOR-002", "location": "warehouse-B", "temperature": 32.4, "humidity": 88.9 }
		

				POST metrics-weather-sensors/_doc
					{
  "@timestamp": "2099-05-06T16:21:15.000Z",
  "sensor_id": "SENSOR-00002",
  "location": "warehouse-B",
  "temperature": 32.4,
  "humidity": 88.9
}
		

Verify setup

To make sure your time series data stream is working, try some GET requests.

View data stream details:

				GET _data_stream/metrics-prod 
		

Check the document count in a time series data stream:

				GET metrics-prod/_count 
		

Query the time series data:

				GET metrics-prod/_search 
					{
  "size": 5,
  "sort": ["@timestamp"]
}