Prune filter does not work

[Juan_David_Jaramillo (Juan David Jaramillo Castro)]

good morning, hope you are well

I have a problem with the "prune" filter in logstash and I have a configuration file in logstash in which the objective is to take the data from filebeat-netflow and go through the pipe in logstash and from there I filter the fields that I need to send to Elasticsearch, I have this configuration, but I do not know if the fields are well mapped according to the syntax that handles netflow for those fields, but the filter does not work, that is to say it does nothing and when running the script is stuck in listening and does not send data.

I need your help with this please, and so feedback about the information of the filter 'prune' with the function 'whitelist_names', as there is no information or consistent examples in the documentation in elastic, do not know how to map correctly filebear-netflow fields and that filter is required as the netflow packs many unnecessary fields that consume significant storage in my elastic cloud, thanks,

input {
  beats {
    port  => 5044
  }
}


filter {
    prune {
        interpolate => true
        whitelist_names => [ "[source][ip]", "[observer][ip]" ]
    }
}



output {
if [observer][ip] == "10.20.248.34" {
  #stdout{ }
  elasticsearch {
    hosts => ["https://xxxxxxxxxxxxx.us-central1.gcp.cloud.es.io:9243"]
    user => "elastic"
    password => "xxxxxxxx"
    index => "ntw"
    }
 }
}

[Badger]

The documentation for the prune filter notes that

This filter currently only support operations on top-level fields, i.e. whitelisting and blacklisting of subfields based on name or value does not work.

See also this post.

[Juan_David_Jaramillo (Juan David Jaramillo Castro)]

Ok, thanks for your answer, but then what filter can I apply to choose which fields I want to send only? so it is not a whitelist, any other script that I can use to indicate and specify to send me only the fields I need?

[Badger]

For your use-case you could try something like this to save the fields you want, and then delete everything

    mutate { add_field => { "[@metadata][source]" => "%{[source][ip]}" "[@metadata][observer]" => "%{[observer][ip]}" } }
    prune {
        whitelist_names => [ "[A][field][that][does][not][exist]" ]
        add_field => { "[source][ip]" => "%{[@metadata][source]}" "[observer][ip]" => "%{[@metadata][observer]}" }
    }

However, I think it more likely you will want

        whitelist_names => [ "@timestamp", "host" ]

[Juan_David_Jaramillo (Juan David Jaramillo Castro)]

filter {
mutate { add_field => { "[@metadata][source]" => "%{[source][ip]}" "[@metadata][observer]" => "%{[observer][ip]}" } }
    prune {
        whitelist_names => [ "@timestamp", "host" ]
        add_field => { "[source][ip]" => "%{[@metadata][source]}" "[observer][ip]" => "%{[@metadata][observer]}" }
    }
}

I tried it but it doesn't work, I run the logstash file and it looks like this:

[Badger]

It looks like it is waiting for new events to arrive.

[Juan_David_Jaramillo (Juan David Jaramillo Castro)]

Yes, but it stays there and does not generate any event and I have waited a long time, because before it did send at once.