Create a datafeed

Auth

Datafeeds retrieve data from Elasticsearch for analysis by an anomaly detection job. You can associate only one datafeed with each anomaly detection job. The datafeed contains a query that runs at a defined interval (frequency). If you are concerned about delayed data, you can add a delay (query_delay') at each interval. By default, the datafeed uses the following query: {"match_all": {"boost": 1}}`.

When Elasticsearch security features are enabled, your datafeed remembers which roles the user who created it had at the time of creation and runs the query using those same roles. If you provide secondary authorization headers, those credentials are used instead. You must use Kibana, this API, or the create anomaly detection jobs API to create a datafeed. Do not add a datafeed directly to the .ml-config index. Do not give users write privileges on the .ml-config index.

Required privileges

Index privileges read
Cluster privileges manage_ml

Path parameters

datafeed_id string required

A numerical character string that uniquely identifies the datafeed. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters.

Query parameters

allow_no_indices boolean · default: true

If true, wildcard indices expressions that resolve into no concrete indices are ignored. This includes the _all string or when no indices are specified.

expand_wildcards union oneOf · default: open

Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values.

Supported values include:

  • all: Match any data stream or index, including hidden ones.
  • open: Match open, non-hidden indices. Also matches any non-hidden data stream.
  • closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.
  • hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.
  • none: Wildcard expressions are not accepted.
Values: all open closed hidden none
ignore_throttled boolean · default: true deprecated

If true, concrete, expanded, or aliased indices are ignored when frozen.

ignore_unavailable boolean · default: false

If true, unavailable indices (missing or closed) are ignored.

Request body

aggregations map string to {} Aggregation

If set, the datafeed performs aggregation searches. Support for aggregations is limited and should be used only with low cardinality data.

frequency union oneOf

The interval at which scheduled queries are made while the datafeed runs in real time. The default value is either the bucket span for short bucket spans, or, for longer bucket spans, a sensible fraction of the bucket span. When frequency is shorter than the bucket span, interim results for the last (partial) bucket are written then eventually overwritten by the full bucket results. If the datafeed uses aggregations, this value must be divisible by the interval of the date histogram aggregation.

Values: -1 0
indices union oneOf

An array of index names. Wildcards are supported. If any of the indices are in remote clusters, the master nodes and the machine learning nodes must have the remote_cluster_client role.

One of: string[] or string
job_id string

Identifier for the anomaly detection job.

max_empty_searches number

If a real-time datafeed has never seen any data (including during any initial training period), it automatically stops and closes the associated job after this many real-time searches return no documents. In other words, it stops after frequency times max_empty_searches of real-time operation. If not set, a datafeed with no end time that sees no data remains started until it is explicitly stopped. By default, it is not set.

query {} Query · default: {"match_all": {"boost": 1}}

The Elasticsearch query domain-specific language (DSL). This value corresponds to the query object in an Elasticsearch search POST body. All the options that are supported by Elasticsearch can be used, as this object is passed verbatim to Elasticsearch.

query_delay union oneOf

The number of seconds behind real time that data is queried. For example, if data from 10:04 a.m. might not be searchable in Elasticsearch until 10:06 a.m., set this property to 120 seconds. The default value is randomly selected between 60s and 120s. This randomness improves the query performance when there are multiple jobs running on the same node.

Values: -1 0
runtime_mappings object

Specifies runtime fields for the datafeed search.

script_fields map string to object

Specifies scripts that evaluate custom expressions and returns script fields to the datafeed. The detector configuration objects in a job can contain functions that use these script fields.

scroll_size number · default: 1000

The size parameter that is used in Elasticsearch searches when the datafeed does not use aggregations. The maximum value is the value of index.max_result_window, which is 10,000 by default.

headers object

Response

application/json
aggregations map string to {} Aggregation
chunking_config object required
mode enum required

If the mode is auto, the chunk size is dynamically calculated; this is the recommended value when the datafeed does not use aggregations. If the mode is manual, chunking is applied according to the specified time_span; use this mode when the datafeed uses aggregations. If the mode is off, no chunking is applied.

Values: auto manual off
time_span union oneOf · default: 3h

The time span that each search will be querying. This setting is applicable only when the mode is set to manual.

Values: -1 0
datafeed_id string required
frequency union oneOf
Values: -1 0
indices string[] required
job_id string required
max_empty_searches number
query {} Query required
query_delay union oneOf required
Values: -1 0
runtime_mappings object
script_fields map string to object
scroll_size number required