Aggregated Index name vs Index name - define in Logstash output

[ssasporta (Sharon Sasporta)]

My Logstash conf output looks like that:

output {
         elasticsearch  {
              hosts => ["localhost:9200"]
              index => "logstash%{[fields][index]}%{+YYYY.MM.dd}"
         }
         stdout { codec => rubydebug }
}

I have aggregation in my filter.

In end of run, I have two Indexces created:

1. Contain the events
2. Contain the aggregations

The events index name: logstashcmserver2017.09.11

The aggregation index name : logstash%{[fields][index]}2017.11.14

Can't I control the aggregation index name? Should it be hard coded?

Thanks
Sharon.

[Christian_Dahlqvist]

Apparently %{[fields][index]} is not set for aggregation records, so you should probably correct that.

[ssasporta (Sharon Sasporta)]

Thanks Christian.

Should I use an if statement in the output for the aggregation ?

Can you refer me to something similar?

I will try to do something and will share the results.

Thanks
Sharon.

[Christian_Dahlqvist]

You could use a conditional to set it if it is not already set, but that depends on whether you have other types of events that could be affected or not.

[ssasporta (Sharon Sasporta)]

Hi Christian,

I created a new output:

output {
         if [aggregation] == "true" {
               elasticsearch  {
                     hosts => ["localhost:9200"]
                     index => "logstashaggregation%{[entity]}%{+YYYY.MM.dd HH:mm}"
               }         
         }
         else  {
                  elasticsearch  {
                          hosts => ["localhost:9200"]
                          index => "logstash%{[fields][index]}%{+YYYY.MM.dd}"
                  }
                  stdout { codec => rubydebug }
         }
}

My aggregation filter looks like that:

                  aggregate {
                          task_id => "%{inputserver}_%{exceptiontype}_%{apiname}"
                          code => " map['totalProcessTime'] ||= 0;
                                    map['counter'] ||= 0;
                                    map['counter'] += 1;
                                    map['totalProcessTime'] += event.get('cputimeinmillisec') 
                          "
                          push_map_as_event_on_timeout  => true
                          timeout => 100
                          timeout_tags => ['_aggregatetimeout']
                          timeout_code => "event.set('avgProcessTime' , ( event.get('totalProcessTime') / event.get('counter') ) );
                                           event.set('aggregation' , true);
                                           event.set('entity' , 'cmserver');
                                           event.set('AggregationFields' , '%{task_id}');
                                          "
                  } 

The aggregated events looks like that:

Three issues:

1. In AggregationFields I want to see the task_id that I set in the aggregation
2. I see the aggregation field with value: true. Why the if in the output doesn't apply and we are in the 'else'
3. entity field contains cmserver. if the 'if' from issue 2 will work, will this value be in the index name?

Thanks
Sharon.

[ssasporta (Sharon Sasporta)]

issue 2 solved
issue 3 worked too.

Issue 1 still not.

 timeout_code => "event.set('avgProcessTime' , ( event.get('totalProcessTime') / event.get('counter') ) );
                 event.set('aggregation' , 'true');
                 event.set('entity' , 'cmserver');
                 event.set('AggregationFields' , %{task_id});

Thanks
Sharon.

[Christian_Dahlqvist]

That sounds like a separate issue, so I would recommend opening a new thread for that. I don't know the aggregation filter very well so will unfortunately not be able to help.

[ssasporta (Sharon Sasporta)]

Thanks