Timeshift functionality in Elasticsearch

Elastic Stack Elasticsearch
1

Hello,

Is there a way, in Elasticsearch, to compare current values against values in another time period .i.e, finding the current value of status 500 errors against the same value 24 hours ago in order to create a trend based graph in a dashboard?

Thanks

1 Like
2

Kibana does this, it's a date aggregation.
If you get what you want working in KB then it's simple to just copy the query you want :slight_smile:

3

so would that be for the date aggregation...

"aggs"=> {
"0"=> {
"date_histogram"=> {
"field"=> "@timestamp",
"interval"=> "day"
}
}
},
"size"=> 0
}

I'm using this query in a ruby script so I'm trying to find the a value for a metric now and what it was 24 hours ago. I tried it using the Kibana dashboard using the date range "gt" => "-25h", "lt" => "-24h" but this didn't work. The metric I'm trying to find is that of value of a metric over a rolling hour and comparing that to the same metric's value over 24 hours ago.

I'd appreciate any assistance.

4

The short answer is that it isn't possible in Elasticsearch yet, but we're working on it. There is a PR open which adds a "serial differencing" aggregation to the new set of Pipeline aggs in 2.0:

You build a histo or date_histo, then embed the new diff agg inside of the histogram and specify what "lag" you want differenced. So in your case, you'd specify a lag of 24hrs. The aggregation then subtracts the current point from the same point 24 hours ago and gives you back the difference.

5

Hi,

so would this be:-

"aggs" => {
"diff"=> {
"date_histrogram" => {
"field"=> "@timestamp",
"interval" => "-24h"
}
}
},
"size"=>0
}

New to elasticsearch so please bear with me!

could I also use the interval of "-1d"?

6

Well, the PR hasn't been merged yet, so the functionality is not available. And it is reliant on new framework (called Pipeline aggregations), which won't be added until Elasticsearch 2.0.

But once the feature is merged, it should work something like this

{
   "aggs": {
      "my_date_histo": {
         "date_histogram": {
            "field": "@timestamp",
            "interval": "hour"
         },
         "aggs": {
            "the_avg": {
               "avg": {
                  "field": "my_field"
               }
            },
            "twenty_four_hours_diff": {
               "diff": {
                  "buckets_path": "the_avg",
                  "lag" : 24
               }
            }
         }
      }
   }
}
7

so what could i use in my code now to get this value?

8

As @polyfractal mentioned, you cannot do this at the moment.

9

Yeah, if you need this value, the only way to do it currently is inside your own application. E.g. pull down the entire histogram, then perform the difference calculation yourself.

10

could i use date math for this query?

i.e. the value from the previous day would be:

"gte"=> "now-2h" and "lte"=> "now-2h/d"

?