How to get percentile value of number field by using logstash pipeline

Elastic Stack Logstash
1

Hi Team,
I am trying to get percentile value of number field from Elasticsearch by using logstash pipeline filter . Could you please help me how to get percentile value of number field from logsatsh pipeline.

2

Hi,

I'm not sure i understand your need, can you share an example of what you would like to do ?

3

Hi grumo,
The requirement is we have one field called "duration" (number type) in elasticsearch. I need to get 75percentile of this duration field in a day and i need to send this value to another team by using logstash pipeline. I am sharing sample logstash pipeline
input {
elasticsearch {
hosts => ["XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXx"]
index => "abc*"
query => '{"query": {
"bool": {
"must": ,
"filter": [

  {
      "exists": {
        "field": "duration"
      }
    },
{"range": {
  "@timestamp": {
    "gte": "now-1d/d","lt": "now/d"
  }
}}]}} }'

}
}

filter {

aggregate {
task_id => "%{duration}"
code => "
" here i have stuck
push_map_as_event_on_timeout => true
timeout_task_id_field => "duration"
timeout => 60
}
}
output
{
Kafka servers
}

4

Hi,
Could someone please help me with this?

5

You can probably have elasticsearch calculate that for you using a percentiles aggregation.

It's a bad match for logstash because to calculate the percentiles you need the entire data set at the same time. To calculate the 75th percentile you would create an array with all the values in it, sort it, and take the value that is entry 0.75*array.length. (And possibly interpolate between that entry and the next one.)