# Aggregated Index name vs Index name - define in Logstash output

**URL:** <https://discuss.elastic.co/t/aggregated-index-name-vs-index-name-define-in-logstash-output/107531>\
**Category:** Logstash\
**Created:** [November 14, 2017, 10:49am UTC](https://discuss.elastic.co/t/aggregated-index-name-vs-index-name-define-in-logstash-output/107531 "2017-11-14T10:49:23Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![ssasporta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ssasporta/32/13695_2.png) [@ssasporta](https://discuss.elastic.co/u/ssasporta)\
**Post date:** [November 14, 2017, 10:49am UTC](https://discuss.elastic.co/t/aggregated-index-name-vs-index-name-define-in-logstash-output/107531/1 "2017-11-14T10:49:23Z")

</div>

My Logstash conf output looks like that:

```
output {
         elasticsearch {
              hosts => ["localhost:9200"]
              index => "logstash%{[fields][index]}%{+YYYY.MM.dd}"
         }
         stdout { codec => rubydebug }
}

```

I have aggregation in my filter.

In end of run, I have two Indexces created:

1. Contain the events
2. Contain the aggregations

The events index name: `logstashcmserver2017.09.11`

The aggregation index name : `logstash%{[fields][index]}2017.11.14`

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/1/c1b1cc819ea4f9deacc09db1e1f3b9a412b47b46.png)

Can't I control the aggregation index name? Should it be hard coded?

Thanks  
Sharon.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 14, 2017, 10:57am UTC](https://discuss.elastic.co/t/aggregated-index-name-vs-index-name-define-in-logstash-output/107531/2 "2017-11-14T10:57:05Z")

</div>

Apparently `%{[fields][index]}` is not set for aggregation records, so you should probably correct that.

---

<div class="post-metadata">

**Author:** ![ssasporta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ssasporta/32/13695_2.png) [@ssasporta](https://discuss.elastic.co/u/ssasporta)\
**Post date:** [November 14, 2017, 12:15pm UTC](https://discuss.elastic.co/t/aggregated-index-name-vs-index-name-define-in-logstash-output/107531/3 "2017-11-14T12:15:51Z")

</div>

Thanks Christian.

Should I use an if statement in the output for the aggregation ?

Can you refer me to something similar?

I will try to do something and will share the results.

Thanks  
Sharon.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 14, 2017, 12:36pm UTC](https://discuss.elastic.co/t/aggregated-index-name-vs-index-name-define-in-logstash-output/107531/4 "2017-11-14T12:36:05Z")

</div>

You could use a conditional to set it if it is not already set, but that depends on whether you have other types of events that could be affected or not.

---

<div class="post-metadata">

**Author:** ![ssasporta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ssasporta/32/13695_2.png) [@ssasporta](https://discuss.elastic.co/u/ssasporta)\
**Post date:** [November 14, 2017, 1:31pm UTC](https://discuss.elastic.co/t/aggregated-index-name-vs-index-name-define-in-logstash-output/107531/5 "2017-11-14T13:31:06Z")

</div>

Hi Christian,

I created a new output:

```
output {
         if [aggregation] == "true" {
               elasticsearch {
                     hosts => ["localhost:9200"]
                     index => "logstashaggregation%{[entity]}%{+YYYY.MM.dd HH:mm}"
               }         
         }
         else {
                  elasticsearch {
                          hosts => ["localhost:9200"]
                          index => "logstash%{[fields][index]}%{+YYYY.MM.dd}"
                  }
                  stdout { codec => rubydebug }
         }
}

```

My aggregation filter looks like that:

```
                  aggregate {
                          task_id => "%{inputserver}_%{exceptiontype}_%{apiname}"
                          code => " map['totalProcessTime'] ||= 0;
                                    map['counter'] ||= 0;
                                    map['counter'] += 1;
                                    map['totalProcessTime'] += event.get('cputimeinmillisec') 
                          "
                          push_map_as_event_on_timeout => true
                          timeout => 100
                          timeout_tags => ['_aggregatetimeout']
                          timeout_code => "event.set('avgProcessTime' , ( event.get('totalProcessTime') / event.get('counter') ) );
                                           event.set('aggregation' , true);
                                           event.set('entity' , 'cmserver');
                                           event.set('AggregationFields' , '%{task_id}');
                                          "
                  } 

```

The aggregated events looks like that:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/c/6c20737cb6a0d1397140ea7ef47c375bd4002f90.png)

Three issues:

1. In AggregationFields I want to see the task\_id that I set in the aggregation
2. I see the aggregation field with value: true. Why the if in the output doesn't apply and we are in the 'else'
3. entity field contains cmserver. if the 'if' from issue 2 will work, will this value be in the index name?

Thanks  
Sharon.

---

<div class="post-metadata">

**Author:** ![ssasporta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ssasporta/32/13695_2.png) [@ssasporta](https://discuss.elastic.co/u/ssasporta)\
**Post date:** [November 14, 2017, 1:49pm UTC](https://discuss.elastic.co/t/aggregated-index-name-vs-index-name-define-in-logstash-output/107531/6 "2017-11-14T13:49:29Z")

</div>

issue 2 solved  
issue 3 worked too.

**Issue 1 still not.**

```
 timeout_code => "event.set('avgProcessTime' , ( event.get('totalProcessTime') / event.get('counter') ) );
                 event.set('aggregation' , 'true');
                 event.set('entity' , 'cmserver');
                 event.set('AggregationFields' , %{task_id});

```

Thanks  
Sharon.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 14, 2017, 2:04pm UTC](https://discuss.elastic.co/t/aggregated-index-name-vs-index-name-define-in-logstash-output/107531/7 "2017-11-14T14:04:38Z")

</div>

That sounds like a separate issue, so I would recommend opening a new thread for that. I don't know the aggregation filter very well so will unfortunately not be able to help.

---

<div class="post-metadata">

**Author:** ![ssasporta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ssasporta/32/13695_2.png) [@ssasporta](https://discuss.elastic.co/u/ssasporta)\
**Post date:** [November 14, 2017, 2:06pm UTC](https://discuss.elastic.co/t/aggregated-index-name-vs-index-name-define-in-logstash-output/107531/8 "2017-11-14T14:06:04Z")

</div>

Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 12, 2017, 2:06pm UTC](https://discuss.elastic.co/t/aggregated-index-name-vs-index-name-define-in-logstash-output/107531/9 "2017-12-12T14:06:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.