What is Query delay and bucket span

Elastic Stack Elasticsearch
1

Hi Team

We are running two jobs of anomaly detection where for one job we are using 10 min of bucket span and for second one we are using 1 hour bucket span. Query delay is same for both jobs i.e. default value.

For both jobs i was getting data latency warning so i changed for one job to look the difference.

Is there any relation between bucket span and Query Delay ?

2

Hello @Aniket_Pant ,

The primary motivation for query delay is to handle delayed data.

To figure out the correct query delay value for your anomaly jobs, do the following:

  1. Create a date_histogram aggregation for the last 24 hours, using the bucket_span value as fixed_interval.
  2. Run this aggregation several times with different delays. For instance, if your bucket span is 10m, you can run the aggregation once, then after 10 minutes, after 20 minutes, etc.
  3. Look at the doc_counts of the results for the same time intervals.
  4. Once the doc_countss for the overlapping histogram buckets no longer change, this gives you the effective query_delay parameter.

Alternatively, if both event time and ingest time for your data are available, you can look at the difference between those to figure out the correct query delay value.

You can also calculate ingest lag as described in this article. This will indicate the query delay you need to set.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.