# Aggregation EPS and Average Elapsed Time

**URL:** <https://discuss.elastic.co/t/aggregation-eps-and-average-elapsed-time/169018>\
**Category:** Logstash\
**Created:** [February 19, 2019, 12:16pm UTC](https://discuss.elastic.co/t/aggregation-eps-and-average-elapsed-time/169018 "2019-02-19T12:16:24Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![rtb](https://avatars.discourse-cdn.com/v4/letter/r/71e660/32.png) [@rtb](https://discuss.elastic.co/u/rtb)\
**Post date:** [February 19, 2019, 12:16pm UTC](https://discuss.elastic.co/t/aggregation-eps-and-average-elapsed-time/169018/1 "2019-02-19T12:16:25Z")

</div>

I need to create an aggregation to provide a count of EPS and Average elapsed time for an operation every second. I have the following but with my test data, I don't get any output.

Below is my filter and I am running within 1 worker, I am using the time stamp and operation to generate the output value for every operation every second. I would expect to see some output but nothing. Any ideas what I'm missing?

grok {  
match =\> { "timestamp" =\> "(T%{HOUR:aggh}:%{MINUTE:aggm}:%{MINUTE:aggs}.)" }  
}  
mutate {  
add\_field =\> { "elapseindex" =\> "%{aggh}%{aggm}%{aggs}" }  
remove\_field =\> ["aggh", "aggm", "aggs"]  
}  
mutate {  
convert =\> ["elapsetime", "integer"]  
convert =\> ["elapseindex", "integer"]  
}  
if [operation] {  
aggregate {  
timeout\_timestamp\_field =\> "timestamp"  
task\_id =\> "%{elapseindex}\_%{operation}"  
code =\> "  
map['avg'] ||= 0;  
map['avg'] += event.get('elapsedtime');  
map['eps\_count'] ||= 0;  
map['eps\_count'] += 1;  
"  
push\_map\_as\_event\_on\_timeout =\> true  
timeout =\> 5  
timeout\_code =\> "  
event.set('elapsed\_avg', (map['avg'] / map['eps\_count']));  
event.set('events\_per\_second', 'eps\_count');  
event.set('agg\_operation', event.get('operation'));  
event.set('Aggregation', true);  
"  
}  
}  
if !['Aggregation'] {  
drop {}  
}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 19, 2019, 1:22pm UTC](https://discuss.elastic.co/t/aggregation-eps-and-average-elapsed-time/169018/2 "2019-02-19T13:22:07Z")

</div>

What does the input look like?

---

<div class="post-metadata">

**Author:** ![rtb](https://avatars.discourse-cdn.com/v4/letter/r/71e660/32.png) [@rtb](https://discuss.elastic.co/u/rtb)\
**Post date:** [February 19, 2019, 2:17pm UTC](https://discuss.elastic.co/t/aggregation-eps-and-average-elapsed-time/169018/3 "2019-02-19T14:17:02Z")

</div>

{  
"elapsedtime" =\> 0,  
"status" =\> "SUCCESSFUL",  
"elapsedtimeunits" =\> "MILLISECONDS",  
"elapseindex" =\> 10101,  
"@timestamp" =\> 2019-02-19T14:11:32.120Z,  
"operation" =\> "SEARCH",  
"timestamp" =\> "2019-01-28T01:01:01.380Z"  
}  
{  
"elapsedtime" =\> 0,  
"status" =\> "SUCCESSFUL",  
"elapsedtimeunits" =\> "MILLISECONDS",  
"elapseindex" =\> 10101,  
"@timestamp" =\> 2019-02-19T14:11:32.120Z,  
"operation" =\> "SEARCH",  
"timestamp" =\> "2019-01-28T01:01:01.381Z"  
}

---

<div class="post-metadata">

**Author:** ![rtb](https://avatars.discourse-cdn.com/v4/letter/r/71e660/32.png) [@rtb](https://discuss.elastic.co/u/rtb)\
**Post date:** [February 19, 2019, 2:52pm UTC](https://discuss.elastic.co/t/aggregation-eps-and-average-elapsed-time/169018/4 "2019-02-19T14:52:02Z")

</div>

I added a date filter to timeastamp and I'm getting farther but this is what I get from logstash:

tap\>, :timeout\_code=\>" \n event.set('elapsed\_avg', (map['avg'] / map['eps\_count']));\n event.set('events\_per\_second', 'eps\_count');\n event.set('agg\_operation', event.get('operation'));\n event.set('Aggregation', true);\n ", :timeout\_event\_data=\>{"avg"=\>0, "@timestamp"=\>2019-02-19T14:49:31.454Z, "@version"=\>"1", "eps\_count"=\>2}}

[ERROR] 2019-02-19 09:49:31.462 [LogStash::Runner] Logstash - org.jruby.exceptions.ThreadKill

[ERROR] 2019-02-19 09:49:31.464 [[main]\>worker0] aggregate - Aggregate exception occurred {:error=\>#\<NameError: undefined local variable or method `map' for #\<LogStash::Filters::Aggregate:0x4a31d81b\>

Did you mean? map\_action

map\_action=

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 19, 2019, 3:18pm UTC](https://discuss.elastic.co/t/aggregation-eps-and-average-elapsed-time/169018/5 "2019-02-19T15:18:13Z")

</div>

That is the output, what does the input look like?

You need to remove the quotes in the test of Aggregation

```
if ![Aggregation] {

```

When the timeout\_code executes map no longer exists, but whatever was in map for this task has been pre-populated

```
event.set('elapsed_avg', (event.get('avg') / event.get('eps_count')));
```

---

<div class="post-metadata">

**Author:** ![rtb](https://avatars.discourse-cdn.com/v4/letter/r/71e660/32.png) [@rtb](https://discuss.elastic.co/u/rtb)\
**Post date:** [February 19, 2019, 3:23pm UTC](https://discuss.elastic.co/t/aggregation-eps-and-average-elapsed-time/169018/6 "2019-02-19T15:23:01Z")

</div>

Yeah, I think I just figured that out... Thanks!

aggregate {  
#timeout\_timestamp\_field =\> "timestamp"  
task\_id =\> "%{elapseindex}\_%{operation}"  
code =\> "  
map['avg'] ||= 0;  
map['avg'] += event.get('elapsedtime');  
map['eps\_count'] ||= 0;  
map['eps\_count'] += 1;  
map['operation'] = event.get('operation');  
"  
push\_map\_as\_event\_on\_timeout =\> true  
timeout =\> 5  
timeout\_code =\> "  
event.set('elapsed\_avg', (event.get('avg') / event.get('eps\_count')));  
event.set('Aggregation', true);  
"  
}

---

<div class="post-metadata">

**Author:** ![rtb](https://avatars.discourse-cdn.com/v4/letter/r/71e660/32.png) [@rtb](https://discuss.elastic.co/u/rtb)\
**Post date:** [February 19, 2019, 4:47pm UTC](https://discuss.elastic.co/t/aggregation-eps-and-average-elapsed-time/169018/7 "2019-02-19T16:47:22Z")

</div>

How does one reference the results of the aggregation in the output?  
It doesn't appear to be %{message}. I am trying to add them to a syslog output.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 19, 2019, 5:45pm UTC](https://discuss.elastic.co/t/aggregation-eps-and-average-elapsed-time/169018/8 "2019-02-19T17:45:18Z")

</div>

Other than @version and @timestamp the events will only have the fields you added to them...

```
{
       "@timestamp" => 2019-02-19T17:44:10.705Z,
      "elapsed_avg" => 0,
              "avg" => 0,
        "eps_count" => 2,
"events_per_second" => "eps_count",
      "Aggregation" => true,
         "@version" => "1",
    "agg_operation" => nil
}
```

---

<div class="post-metadata">

**Author:** ![rtb](https://avatars.discourse-cdn.com/v4/letter/r/71e660/32.png) [@rtb](https://discuss.elastic.co/u/rtb)\
**Post date:** [February 19, 2019, 8:27pm UTC](https://discuss.elastic.co/t/aggregation-eps-and-average-elapsed-time/169018/9 "2019-02-19T20:27:41Z")

</div>

Thanks, That is what I figured... This is to get data into splunk without impacting the licensing cost too bad... lol...

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 19, 2019, 8:27pm UTC](https://discuss.elastic.co/t/aggregation-eps-and-average-elapsed-time/169018/10 "2019-03-19T20:27:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.