LS5.3: Variable codec setting

Elastic Stack Logstash
1

Hi!
I wanted to optimize my Logstash config. The old one looked like this:

output {
  if ("json_extracted" in [tags]) {
    elasticsearch {
      codec => "json"
      hosts => <%= $es_hosts.prefix('"').suffix('"') %>
      index => '%{elk_input}-%{+YYYY.MM.dd}'
    }
  } else {
      elasticsearch {
        codec => "plain"
        hosts => <%= $es_hosts.prefix('"').suffix('"') %>
        index => '%{elk_input}-%{+YYYY.MM.dd}'
      }
  }
...
}

This pattern is repeated multiple times. You can see the only difference is the codec setting (json vs. plain). I thought I could use a variable for that, but did not find a way how to set one. Luckily I found the @metadata field. So I changed my code to this:

filter {
  if ("json_extracted" in [tags]) {
    mutate { add_field => { "[@metadata][codec]" => "json" } }
  } else {
    mutate { add_field => { "[@metadata][codec]" => "plain" } }
  }
}
output {
  elasticsearch {
    codec => "%{[@metadata][codec]}"
    hosts => <%= $es_hosts.prefix('"').suffix('"') %>
    index => '%{elk_input}-%{+YYYY.MM.dd}'
  }
...
}

What it should do is decide if json or plain is used, store that in @metadata.codec and use it in the output on field codec.
Well, that doesn't work. Is the idea totally wrong or did I do a syntax error which I have missed?

Thanks for your help,
Alex

2

There's never a need to modify the codec option of the elasticsearch output. Just leave it alone and things will work fine.

3

Thanks! I thought I need to set it to json if I want to write messages in json. Good to know that this is not true. I will remove it.

Nevertheless I would like to know if there are variables which I can set and get in Logstash config. It seems that not every setting supports the %{} syntax. Is that correct?

4

Nevertheless I would like to know if there are variables which I can set and get in Logstash config.

You already know about the @metadata field and the rest of your configuration should work, it's just that the use of codec doesn't make any sense (plus the codec option probably don't support that kind of string interpolation).

It seems that not every setting supports the %{} syntax. Is that correct?

That's correct. Unfortunately it's not documented where it's supported and where it isn't.

1 Like
5

Perfect! Thanks for your answer.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.