Aggregate filter dilemma

Elastic Stack Logstash
1 
	Hi all,

hoping to get some feedback on this as it took me 3 days to resolve which I think is potentially a knowledge gap on my side or a BUG.

In summary:

  1. Two logs, I want to aggregate using the aggregate filter. One field in common between both logs
    if "log1" in [tags] {

    		grok {
				match => ["message", "(?<log1_KEY>(?<=log1key:).*?(?=\s,))"]
			}
		
		
		aggregate {
		 task_id => "%{log1_KEY}"
		 .....
		 
		 }
		 
		 }
		 	 
		 
		if "log2" in [tags] {

    		grok {
				match => ["message", "(?<log2_KEY>(?<=log2key:).*?(?=\s,))"] # log2 key would be same as log1 key as they are correlated
			}
		
		
		aggregate {
		 task_id => "%{log2_KEY}"
		 .....
		 
		 }
		 
		 }

The issue here is that aggregate filter treats both log1_key and log2_key differently even though both contain the same value. They only way I got this work is by using the same name for key id such as log_key.

So in summary I see the following when using aggregate fitler:

log1_key = 123 log2_key = 123

log1_key == log2_key ---> aggregate filter fails to match. log_key == log_key --> works

I am confused as to why this behaviour exists when the underlying values are the same?

2

That is expected. The array of map elements is indexed by both the value of task_id and the sprintf'd value of task_id. If the names of the two fields are different then they will be treated differently, even if the contents of the two fields are the same.

3

thanks for the quick response. Is there any particular reason for this as this goes against all programming concepts (as far as I can think)?

4

I cannot speak to the programmer's thinking when they designed it, but to me it makes sense.

5

Many thanks for the quick reply.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.