Add additional field (count of another field)

[treknado]

Hello,

i have documents with multiline fields in my.field and need to count the number of received filenames.
@Fabio-sama suggested me in another thread (which was in the wrong section) something like that:

{
"my_field":[
{
"filename": "test1.json",
"type" : "File"
},
{
"filename": "test2.json",
"type" : "File"
}
],
"number_of_filenames": 2
}

How can I do that in logstash in a simple way ?

@Fabio-sama:
is that my_field field always a list of objects? And is filename present in each of them? I'm asking if you have to make any check on the single objects or if you blindly get the size of the list.
It is a list of objects OR is an empty field. So it is empty OR the structure i posted above

best regards,
treknado

[Fabio-sama (Fabio)]

By empty you mean an empty list? an empty string? Or nothing at all (field is not present)?

Please post an example of a document where that field is "empty".

[Fabio-sama (Fabio)]

Well, the following should work anyway:

ruby {
  code => "
      my_field = event.get('my_field')
      number_of_filenames = my_field.nil? ? 0 : my_field.size
      event.set('number_of_filenames', number_of_filenames)
  "
}

Put it in the filter section of the LS pipeline

[treknado]

i am a little bit confused. It looks like kibana created some new fields (which are not visible in discover) in my index pattern:

my_field.name
my_field.type

I checked via the Elastic API if the value of my_field is a string and found out: it is not a string.
Looks like an inner object.

so the answer is:
the field is present, and the field is not mapped.

[Fabio-sama (Fabio)]

As suggested there, you have to refresh your Index Pattern to see the mapping of the new field. Also, if it's not automatically recognized as a integer, use a template to set the right mapping.

Btw, which version of the stack are you using?

[treknado]

yeah, i tried to do that. the index-pattern-refresh did not change anything. Also delete the index pattern and create a new index pattern did not change anything.

i am using 7.4.2

[Fabio-sama (Fabio)]

Wait, the my_field field should be the one that was already there.

Is there the new numeric field number_of_filenames? That is the field you have to make the sum on.

[treknado]

ah, i did not read the ruby code you posted above. will try it out today.

But i have still the issue, that my_field is not mapped properly.
when the list in my_field is [], the mapping from ES is strange.

[Fabio-sama (Fabio)]

Ok so, do you confirm me you can now do what you were trying to do when you started this thread (or even the previous one in the Kibana section), that is making the sum of all the items of a specific field?

For this second mapping problem, can you share a sample of a document with a non-empty my_field field and another with a empty one?

Maybe also a snapshot of the Discover section to see the error in mapping.