Hi,
I have a problem with Logstash Aggregate filter. The exact same filter works in Logstash 7.5.0 but not in Logstash 6.6.1 and I can't find out why. The Aggregate filter version in 7.5.0 is 2.9.1 and in 6.6.1 it is 2.9.0 - and it doesn't seems as it happened so much between these versions.
The input, filters and output,
input {
file {
path => "/mnt/testlogs/*"
}
}
# Before cloning:
# Generate uuid, in order to match the original and the cloned event.
# If there is another field to match the events it can be used as well.
filter {
ruby {
code => "event.set('uuid', rand(36**10).to_s(36))"
}
}
# Cloning the event
filter {
clone {
clones => ["cloned"]
}
}
filter {
# Adding a field to the cloned event
if [type] == "cloned" {
mutate {
add_field => { "origin.hostname" => "myhost" }
}
mutate {
copy => { "origin.hostname" => "hfolder" }
}
# In the cloned event,
# Creating an aggregate map that is shared between events that have the same task_id = uuid.
# Adding the first and second field to the map
aggregate {
task_id => "%{uuid}"
code => "map['hfolder'] ||= event.get('hfolder')"
}
# Clean up - remove uuid
mutate { remove_field => ["uuid"] }
# In the original event
# Declare the aggregate map with the same task_id as the cloned event.
# Copy the fields from the aggregate map to the event.
#
} else {
aggregate {
task_id => "%{uuid}"
code => "event.set('hfolder', map['hfolder'])"
# Delete the aggregate map from memory, as it is no longer needed.
end_of_task => true
}
# Clean up - remove uuid
mutate {
remove_field => ["uuid"]
}
# The mutate filter below is here only for debugging purposes. It can be deleted.
mutate {
add_field => { "iscloned" => "no" }
}
}
}
output {
if [type] == "cloned" {
stdout { codec => rubydebug }
} else {
stdout { codec => rubydebug }
}
}
Result in Logstash 6.6.1,
"iscloned" => "no",
"@version" => "1",
"message" => "(deleted)",
"path" => "/mnt/testlogs/test.log",
"@timestamp" => 2020-01-08T14:02:50.743Z,
"hfolder" => nil
"@version" => "1",
"host" => "216d984a3495",
"message" => "(deleted)",
"origin.hostname" => "myhost",
"type" => "cloned",
"path" => "/mnt/testlogs/test.log",
"@timestamp" => 2020-01-08T14:02:50.743Z,
"hfolder" => "myhost"
Result in Logstash 7.5.0,
"host" => "9c76c3424d39",
"type" => "cloned",
"path" => "/mnt/testlogs/test.log",
"hfolder" => "myhost",
"message" => "(deleted)",
"@version" => "1",
"origin.hostname" => "myhost",
"@timestamp" => 2020-01-08T13:18:00.424Z
"@timestamp" => 2020-01-08T13:18:00.380Z,
"path" => "/mnt/testlogs/test.log",
"hfolder" => "myhost",
"message" => "(deleted)",
"iscloned" => "no",
"@version" => "1",
"host" => "9c76c3424d39"
pipeline.workers is set to 1 in both environments.
In 6.6.1 the original (not clone) event "hfolder" gets "nil" as value. In 7.5.0 "myhost" as expected.
I am grateful for all suggestions and input.
/Bjorn
