How to write to a multiple indexes using single elasticsearch output?

Elastic Stack Logstash
1

Hey folks, i'm pretty new guy for ELK stack, so don't judge me too much.

So we have a basic fresh ELK setup v 7.6.2 with basic licence enabled.

We are using filebeat to gather logs and send them to logstash.

After that we need to write those logs to a 3 different indexes depending on field (field.env:[stage|preprod|prod])

We created this field using filebeat config with entry like this:

fields:
 env: stage

ILM policy is on by default (but i assume we can live without it in future)

So we tried something like this - https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#_writing_to_different_indices_best_practices

And my logstash config was smthing like that:

input {
  beats {
    port => 5044
    host => "******"
  }
}

filter {
      if [field.env] == stage {
        mutate { add_field => { "[@metadata][logstash-2020.04.06-000001]" => "stage-%{+YYYY.MM}" } }
      } else if [field.env] == "preprod" {
        mutate { add_field => { "[@metadata][logstash-2020.04.06-000001]" => "preprod-%{+YYYY.MM.dd}" } }
      } else {
        mutate { add_field => { "[@metadata][logstash-2020.04.06-000001]" => "prod-%{+YYYY}" } }
      }
}

output {
    elasticsearch {
    hosts => ["127.0.0.1:9200"]
    user => "elastic"
    password => "******"
    }
}

logstash-2020.04.06-000001 - is a default index created by logstash

After creating this config file all stack was restarted, all services were up and running without any errors.

But, logstash was continuing to write all logs to default index, witch is logstash-2020.04.06-000001.

So i need an advise:

  1. what logstash config need to look like to create and write to a different indexes with mentioned requirements?
  2. what do we need to turn off to be able to do that, like ILM or smthng?

Thank you for your help!!!

2

That should be

if [fields][env] == "stage" {
3

Will try it asap, anything else?
May be something i need to modify in "output" section?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.