Aggregate fields to an array fields

Elastic Stack Logstash
1

Hi,

I have some output here from my logstash filter:

{
   "taskid" => "123",
   "host" => "ubuntu"
},
{
  "taskid" => "123",
  "host" => "centos"
}

is it possible to aggregate that field to be like this:

{
  "taskid" => "123",
  "host" => [ "ubuntu", "centos" ]
}

Please for the answer.
Thanks.

2

Yes, that could be done with an aggregate filter. You might use something like example 4 in the documentation, but depending on your data you might need something more like example 3.

3

Hi Badger,

Thanks for the reply.

I have tried that and it's work. But now the problem is,

When the data is like this,

{
   "taskid" => "123",
   "host" => "ubuntu"
},
{
  "taskid" => "123",
  "host" => "centos"
},
{
  "taskid" => "123",
  "host" => "ubuntu"
},
{
  "taskid" => "123",
  "host" => nil,
  "ip" => "1.2.3.4"
}

The output become like this,

{
  "taskid" => "123",
  "host" => [ "ubuntu", "centos", "ubuntu", nil ],
  "ip" => "1.2.3.4"
}

is there any way to prevent the multiple value and the empty value?

I can't drop the empty host field because i need the ip field.

4

If you are currently doing

map["host"] << event.get("host")

you could change that to

h = event.get("host")
if h
   map["host"] << h
end
5

Thanks for the answers.

I've tried that and it works for the empty field, but not for the multiple values.

still getting output like this,

{
  "taskid" => "123",
  "host" => [ "ubuntu", "centos", "ubuntu" ],
  "ip" => "1.2.3.4"
}
6

I do not understand what you mean about multiple values. What do you want the event to look like?

7

oh wait,
i mean duplicate value? is that right?

what i want is, the host field does not have any duplicate value.
the output should be like this,

"host" => [ "ubuntu", "centos" ]

not like this,

8

Yes, you can remove duplicates. How to do that will depend on what your existing aggregate looks like. Either not add them or remove them at the end.

You might need

if h && ! maps["host"].include? h
    map["host"] << h

If your aggregate is currently doing event.set("host", maps["host"]) then you could change it to

event.set("host", maps["host"].uniq)
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.