Logstash Aggregate filter plugin works in 7.5.0, not in 6.6.1

Elastic Stack Logstash
1

Hi,

I have a problem with Logstash Aggregate filter. The exact same filter works in Logstash 7.5.0 but not in Logstash 6.6.1 and I can't find out why. The Aggregate filter version in 7.5.0 is 2.9.1 and in 6.6.1 it is 2.9.0 - and it doesn't seems as it happened so much between these versions.

The input, filters and output,

input {
  file {
    path => "/mnt/testlogs/*"
  }
}
 
# Before cloning: 
# Generate uuid, in order to match the original and the cloned event. 
# If there is another field to match the events it can be used as well.  
 
filter {
  ruby {
    code => "event.set('uuid', rand(36**10).to_s(36))"
  }
}
 
# Cloning the event
filter {
  clone {
    clones => ["cloned"]
  }
}
 
filter {
# Adding a field to the cloned event
  if [type] == "cloned" {
    mutate {
      add_field => { "origin.hostname" => "myhost" }
    }
    mutate {
      copy => { "origin.hostname" => "hfolder" }
    }
# In the cloned event,
# Creating an aggregate map that is shared between events that have the same task_id = uuid.
# Adding the first and second field to the map 
    aggregate {
      task_id => "%{uuid}"
      code => "map['hfolder'] ||= event.get('hfolder')"
    }
# Clean up - remove uuid
    mutate { remove_field => ["uuid"] }
      
# In the original event
# Declare the aggregate map with the same task_id as the cloned event.
# Copy the fields from the aggregate map to the event.
#    
  } else { 
    aggregate {
      task_id => "%{uuid}"
      code => "event.set('hfolder', map['hfolder'])"
# Delete the aggregate map from memory, as it is no longer needed. 
      end_of_task => true
    }
      
# Clean up - remove uuid
    mutate { 
      remove_field => ["uuid"]   
    }
# The mutate filter below is here only for debugging purposes. It can be deleted.
    mutate {
      add_field => { "iscloned" => "no" }
    }
  }
}
 
output {
  if [type] == "cloned" {
    stdout { codec => rubydebug }
  } else {
    stdout { codec => rubydebug }
  }
}

Result in Logstash 6.6.1,

"iscloned" => "no",
"@version" => "1",
"message" => "(deleted)",
"path" => "/mnt/testlogs/test.log",
"@timestamp" => 2020-01-08T14:02:50.743Z,
"hfolder" => nil

"@version" => "1",
"host" => "216d984a3495",
"message" => "(deleted)",
"origin.hostname" => "myhost",
"type" => "cloned",
"path" => "/mnt/testlogs/test.log",
"@timestamp" => 2020-01-08T14:02:50.743Z,
"hfolder" => "myhost"

Result in Logstash 7.5.0,

"host" => "9c76c3424d39",
"type" => "cloned",
"path" => "/mnt/testlogs/test.log",
"hfolder" => "myhost",
"message" => "(deleted)",
"@version" => "1",
"origin.hostname" => "myhost",
"@timestamp" => 2020-01-08T13:18:00.424Z

"@timestamp" => 2020-01-08T13:18:00.380Z,
"path" => "/mnt/testlogs/test.log",
"hfolder" => "myhost",
"message" => "(deleted)",
"iscloned" => "no",
"@version" => "1",
"host" => "9c76c3424d39"

pipeline.workers is set to 1 in both environments.

In 6.6.1 the original (not clone) event "hfolder" gets "nil" as value. In 7.5.0 "myhost" as expected.

I am grateful for all suggestions and input.

/Bjorn

2

In 7.5.0, if you set '--pipeline.java_execution false' on the command line does it stop working? If so, you are relying on java re-ordering your events (which is a bug that will one day get fixed).

1 Like
3

Thanks, I will try that and see if it stops working.

UPDATE

I did the opposite and added "pipeline.java_execution: true" in 6.6.1 and now it works. It seems as it was implemented in 6.5 and set to false as default.

Thank you very much! :slight_smile:

/Bjorn

4

OK, but be aware that you are relying on a bug that will one day get fixed.

1 Like
5

Yes, I will. But in this special environment we are going to use 6.6.1 for the foreseeable future. No updates at all.

Thanks again.

/Bjorn

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.