Apply index, if available

[johann]

Hi,

if I have an beats input from journalbeat to logstash, how can I use the index which is already set?

The beats input is coming from journalbeat -> logstash -> elasticsearch.

Journalbeat output: output.logstash with index option.
In my Logstash config I have a few if-else rules.

Is there a way to set this index from journalbeat if there is any?

For example:

if [input] == journalbeat {
    if isset [index] {
      // keep index
    } else {
    index => "notset-%{+YYYY.MM.dd}"
    }
}

Thanks in advance!

Cheers,

johann

[Badger]

@johann

if isset [index] { // keep index } else { index => "notset-%{+YYYY.MM.dd}" }

That should be

if ! [index] {
     mutate { add_field { index => "notset-%{+YYYY.MM.dd}" } }
}

[johann]

Thank you, the problem is, the logstash output.elasticsearch sets an default index ( * Default value is "logstash-%{+YYYY.MM.dd}"), if the index option is not set... (https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-index)

So something like

output {
[...]
else if [_index] {
  elasticsearch {
    hosts => "elasticsearch:9200"
    index => [_index]
  }
}

should be necessary

[Badger]

I am suggesting you add that in the filter so that index is always set when you get to the output.

[johann]

Isn't the index is set by journalbeat already if I use the journalbeat output.logstash?
Should be if I understand https://www.elastic.co/guide/en/beats/journalbeat/6.7/logstash-output.html#logstash-index correct

[Badger]

RIght, that's why I made adding index conditional upon index not being set.

[johann]

Unfortunately the following is not working If I use an other field than index, it works as aspected...

output {
[...]
    else if [index] or [_index] {
      elasticsearch {
                    hosts => "host"
                    index => "auto-%{index}"
      }
    } 
}