Default values for @metadata?

Elastic Stack Logstash
1

Hi,

is there a recommended way to set some default @metadata fields in Logstash for all documents? I'm using this logstash output config to "shard" different types of logs to their own indices as ESv6 no longer supports more than one type of documents per index

output {

  elasticsearch {
        hosts => ["10.0.0.3:9200", "10.0.0.18:9200", "10.0.0.26:9200"]
        index => "%{[@metadata][log_prefix]}-%{[@metadata][index]}-%{+YYYY.MM.dd}"
  }
}

For documents that to not have these metadata fields set I get these indices

$ curl -s http://localhost:9200/_cat/indices | grep meta
green open %{[@metadata][log_prefix]}-%{[@metadata][index]}-2018.06.18 -N6-8xHgTK6YxihD0euTJw 5 1 7002784   0   2.6gb   1.3gb
green open %{[@metadata][log_prefix]}-%{[@metadata][index]}-2018.05.23 fouzYG2ZQS2-OmGneendKg 5 1   12354   0   9.3mb   4.6mb

which is not quite what I had planned...

One way would be to use the alter filter plugin at the top of the filter chain.

filter {
  alter {
    add_field => { "%{[@metadata][log_prefix]}" => "my_prefix" }
    add_field => { "%{[@metadata][index]}" => "my_index" }
  }
}

Any better way to do it?

Cheers,
AB

2

Any better way to do it?

No.

3

Cool, thanks @magnusbaeck

4

And what if I already set metadata on the Logstash input and only want to add if it does not exist :blush:

  if "[@metadata][log_prefix]" {
    WHAT GOES HERE IF I JUST WANT TO EXIT THE IF??
  }
  else {
    alter {
      add_field => { "%{[@metadata][log_prefix]}" => "my_prefix" }
    }
  }

or is it possible to do something like

  if no "[@metadata][log_prefix]" {
    alter {
      add_field => { "%{[@metadata][log_prefix]}" => "my_prefix" }
    }
 }
5 
if ! "[@metadata][log_prefix]" {

https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#conditionals

1 Like
6

That's nice :slight_smile:

Thank you @magnusbaeck once again. I was reading that exact documentation yesterday but it did not occur to me that you could do that :man_facepalming:

7

Got this working and just wanted to paste the syntax that does what I wanted it to do

  if ! [@metadata][log_prefix] {
    alter {
      add_field => {
        "[@metadata][log_prefix]" => "fix"
      }
    }
  }

No double quote after if !

Very happy with the outcome. Works exactly as I had hoped.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.