Translate filter multiple values in same label ECS field

I am looking to populate the [labels][feed] field (labels is the ECS field) with contents from two separate CSV fields as below. I am hoping to have it as an array.
Is this a correct approach? If not, what would be the way to have different values populate the same field as an array

Expected in Elastic Output:
labels.feed ---> [feed_value1, feed_value2]

translate {
        field => "[destination][ip]"
        dictionary_path => "/enrich/feed1.csv"
        exact => false
        destination => "[labels][feed]"
        fallback => ""
        add_tag => [ "import_feed1_ok"]
    translate {
        field => "[source][ip]"
        dictionary_path => "/enrich/feed2.csv"
        exact => false
        destination => "[labels][feed]"
        fallback => ""
        add_tag => [ "import_feed2_ok"]

Thanks in advance!

No, the translate filter is a no-op if the destination field already exists, unless the overwrite option is enabled, and it that case, as you might expect, it overwrites the existing value.

What you could try (I have not tested it) is to use

destination => "[@metadata][feed1]"

for one feed, and

destination => "[@metadata][feed2]"

then use mutate

mutate { add_field => { "[labels][feed]" => "%{[@metadata][feed1]}" } }
mutate { add_field => { "[labels][feed]" => "%{[@metadata][feed2]}" } }

I would expect that to work because add_field takes care to convert the field to an array.

Thanks @Badger !

I assume below would work too right?

mutate { 

add_field => { "[labels][feed]" => "%{[@metadata][feed1]}" }
add_field => { "[labels][feed]" => "%{[@metadata][feed2]}" } 


Had another question on the similar topic as the docs are not very clear (to me)
What if i have a CSV with multiple values with the first column as the key and the rest of the columns having values that i want to add to various other fields.

What would be the way to use the translate filter for this scenario?

docs states

It is possible to provide multi-valued dictionary values. When using a YAML or JSON dictionary, you can have the value as a hash (map) or an array datatype. When using a CSV dictionary, multiple values in the translation must be extracted with another filter e.g. Dissect or KV.
Note that the fallback is a string so on no match the fallback setting needs to formatted so that a filter can extract the multiple values to the correct fields.

Probably, but I never supply multiple copies of an option to a filter. logstash will combine them, almost always in the way you would expect. But only almost always. Over the years I have seen a couple of cases where it did something really unexpected.

To give a couple of examples of multi-valued dictionaries...


foo: { "a": 1,  "b": 2,  "c": 3 }
bar: { "a": 2,  "b": 4,  "d": 8 }

You would look up foo or bar in the dictionary and the parse the translation with a json filter.

csv and kv

foo,a=1 b=2 c=3
bar,a=4 b=6 d=8

csv and JSON are not a good mix because there are two uses for commas.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.