Logstash add field metadata not working

Elastic Stack Logstash
1

I have tried a number of different solutions and cannot get this to work. What am I missing?

https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html

filter {
  mutate { id => "add-pipeline"
           add_field => { "metadata-pipeline" => "%{[@metadata][pipeline]}" }}
}
filter {
  mutate { id => "add-pipeline"
           add_field => { "metadata-pipeline" => "" }}
  mutate {
        replace => { "metadata-pipeline" => "%{[@metadata][pipeline]}" }
      }
}
2

Does the file [@metadata][pipeline] exist? If it does then either the add_field or the replace should work.

3

It does exist and is used for conditional elasticsearch outputs so I know there is a value to it. Here is what I get in the document as a string literal "%{[@metadata][pipeline]}"

 if [@metadata][pipeline] {
  elasticsearch {
    id => "beats-pipeline"
    hosts => ["https://siem-elasticsearch-01:9200"]
    ilm_enabled => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}"
    pipeline => "%{[@metadata][pipeline]}"
    user => beats_ingest
    password => *******
    cacert => "/etc/logstash/ca.crt"
    ssl => true
  }
  }  else {
4

Correction, This is working when there is a pipeline such as "filebeat-7.8.1-elasticsearch-gc-pipeline". When the pipeline is not available it defaults to the string literal so I added some conditional procpessing to mutate for my needs:

filter {
  if [@metadata][pipeline] {
    mutate {
           id => "add-pipeline"
           add_field => { "metadata-pipeline" => "%{[@metadata][pipeline]}" }
    }
  }
  else {
    mutate {
           id => "no-pipeline"
           add_field => { "metadata-pipeline" => "None" }
    }
  }
}
5

It does not help you now, but note that the output was modified yesterday so that if the sprintf evaluates down to "" then it is ignored. As of now, that is going to pass "None" through to elasticsearch, which may complain.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.