Multiple output options in logstash.conf

Elastic Stack Logstash
1

Hi,
I'm trying to come up with a logstash.conf to seperate the indexing method between 2 different sources. One will be daily index and other will be monthly index. I came up with something like this but it didn't work.

output {
    elasticsearch {
        hosts => ["https://odfe-node1:9200"]
        index => "%{tag}=x-%{+YYYY.MM.dd}"
        ssl => true
        ssl_certificate_verification => false
        user => *****
        password => *****
        ilm_enabled => false
    }
    elasticsearch {
        hosts => ["https://odfe-node1:9200"]
        index => "%{tag}=y-%{+YYYY.MM}"
        ssl => true
        ssl_certificate_verification => false
        user => *****
        password => *****
        ilm_enabled => false
    }
    stdout{
    }
}

With this output i didn't even receive logs.

output {
    elasticsearch {
        hosts => ["https://odfe-node1:9200"]
        index => "%{tag=x}-%{+YYYY.MM.dd}"
        ssl => true
        ssl_certificate_verification => false
        user => *****
        password => *****
        ilm_enabled => false
    }
    elasticsearch {
        hosts => ["https://odfe-node1:9200"]
        index => "%{tag=y}-%{+YYYY.MM}"
        ssl => true
        ssl_certificate_verification => false
        user => *****
        password => *****
        ilm_enabled => false
    }
    stdout{
    }
}

And with this one it created two indices(example: 1 daily and 1monthly indice for both x and y sources) for the same source. How can i solve this issue?

Thanks.

2

You can use if else condition for outputs

  if "x" in [tag] {
    elasticsearch {
      index => "index_name"
    }
  }

OR

  if [tag] == "x" {
    elasticsearch {
      index => "index_name"
    }
  }
3

Do you mean like this?

output {
   if "tag" == "x" {
      elasticsearch {
          hosts => ["https://odfe-node1:9200"]
          index => "%{tag}-%{+YYYY.MM.dd}"
          ssl => true
          ssl_certificate_verification => false
          user => *****
          password => *****
          ilm_enabled => false
    }
}

I see this and a couple more like this one below in logstash logs:

{


          "image_id" => "sha256:ae2feff98a0cc5095d97c6c283dcd33090770c76d63877caa99aefbbe4343bdd",


    "container_name" => "x",


          "@version" => "1",


             "level" => 6,


           "message" => "/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh",


           "created" => "2020-12-30T10:49:17.179464622Z",


       "source_host" => "192.168.0.186",


           "version" => "1.1",


        "image_name" => "nginx:latest",


           "command" => "/docker-entrypoint.sh nginx -g daemon off;",


               "tag" => "x",


              "host" => "localhost.localdomain",


        "@timestamp" => 2020-12-30T11:53:35.517Z,


      "container_id" => "4db2ba824b555e56ba574e8dcd238c37aa1568e22936582703a47b539f4b0c43"


}

But no indices are created.

4

Try

output {
   if [tag] == "x" {
      elasticsearch {
          hosts => ["https://odfe-node1:9200"]
          index => "%{tag}-%{+YYYY.MM.dd}"
          ssl => true
          ssl_certificate_verification => false
          user => *****
          password => *****
          ilm_enabled => false
    }
}
5

It worked!! Thank you so much sir. I have one last question. This was merely a test environment to check multiple output conditions. In production we have a much larger indice pool and would like to make some adjustments to indexing with a similar output config. Is it possible to define multiple indices to a single if statement using comma as a seperator? Pretty much like this:

if [tag] == "x", "y", "z", "t", "u" ... {

Thanks.

6

What you can do, create a new metadata field "[@metadata][target_index]" in your filters processing`and have only one output as follow

index => "%{[@metadata][target_index]}"
7

I didn't quite understand this. I dont want to have one output, i need to seperate certain indices as daily or monthly. Also i couldn't picture how the line you gave fits in a config since i'm pretty bad at the filter section. Do you have a complete config as example so i can make sense somehow? Or check my conf out and edit that if you'd like.

input {
    gelf {
        port => 12201
       }
}
filter {
}

output {
   if [tag] == "x" {
      elasticsearch {
          hosts => ["https://odfe-node1:9200"]
          index => "%{tag}-%{+YYYY.MM.dd}"
          ssl => true
          ssl_certificate_verification => false
          user => *****
          password => *****
          ilm_enabled => false
    }
}
   if [tag] == "y" {
      elasticsearch {
          hosts => ["https://odfe-node1:9200"]
          index => "%{tag}-%{+YYYY.MM}"
          ssl => true
          ssl_certificate_verification => false
          user => *****
          password => *****
          ilm_enabled => false
    }
}
    stdout{
    }
}

What happens if i have 4 indices a, b, c and d and i set a, b as daily and c, d as monthly? And what must i do to add an "e" indice when this logstash is running? Does metadata part also cover this? Or must i add "e" to the conf and restart logstash? Sorry for bombarding you with questions, i'm just trying to fully grasp this.
Thanks.

8

Something like this,

input {
    gelf {
        port => 12201
       }
}

filter {
      if [tag] == "x" {
        mutate { add_field => { "[@metadata][target_index]" => "%{tag}-%{+YYYY.MM.dd}" } }
      } else if [tag] == "y" { {
        mutate { add_field => { "[@metadata][target_index]" => "%{tag}-%{+YYYY.MM}" } }
      } else {
        mutate { add_field => { "[@metadata][target_index]" => "unknown-%{+YYYY}" } }
      }
    }

output {
      elasticsearch {
          hosts => ["https://odfe-node1:9200"]
          index => "%{[@metadata][target_index]}"
          ssl => true
          ssl_certificate_verification => false
          user => *****
          password => *****
          ilm_enabled => false
    }
}
9

Everything worked perfectly, thank you so much for your help. I wish i was able to select multiple replies as solutions.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.