How to use if else statments in logstash output pipline?

Elastic Stack Logstash
1

Dear ES folks,

Here is my request, I want to split the index by tags and environment.

Can someone help me out?

The logical like this：

"system" in [tags] => index => "system-%{+YYYY.MM}"
"nginx" in [tags] => index => "nginx-%{+YYYY.MM}"
not ("nginx" and "system") in [tags] and [env]="production" =>other-production-%{+YYYY}
not ("nginx" and "system") in [tags] and [env]="test" =>other-test-%{+YYYY}

code:

#logstash for tlsbeat
input {
 tcp {
   port => 5045
 }
}


filter {

 if "_jsonparsefailure" in [tags] {
       drop { }
 }
 
 json {
   source => "message"
 }
 
 mutate {
   remove_field =>  ["message"]
   add_field => { "[@metadata][index_prefix]" => "%{agent}-%{env}-%{service}" }  
   remove_field =>  ["agent","env","host","port","service"]
 }
 
}

output {
 
 if "system" in [tags] {  
   elasticsearch {
     hosts => ["elkdbtest01.tls.ad"]
     template_overwrite => false
     manage_template => false
     index => "%{[@metadata][index_prefix]}-%{+YYYY.MM}"
     sniffing => false

   }      
 } 
 
 else if "nginx" in [tags] {  
   elasticsearch {
     hosts => ["elkdbtest01.tls.ad"]
     template_overwrite => false
     manage_template => false
     index => "%{[@metadata][index_prefix]}-%{+YYYY}"
     sniffing => false
   }      
 }



}
1 Like
2

Why not set the [@metadata][index_prefix] field to the correct index prefix in the filter block and then use a single Elasticsearch output as all parameters except the index name appear to be the same?

3

Hey @Christian_Dahlqvist,

I used [@metadata][index_prefix] for following tags
"system" in [tags] => index => "system-%{+YYYY.MM}"
"nginx" in [tags] => index => "nginx-%{+YYYY.MM}"
not nginx and system => => index => "other -%{+YYYY.MM}"

I can't use single one elasticsearch output. It's hard to say, It's a little more complicated than normal one.
if I use one ES output, It will generate a dozen of indices, I just want to generate 3 kinds of type index,

System, Nginx And others put them as a single one.

can you suggest or give me an example?

4

What Christian means is, set [@metadata][index_prefix] to the index prefix you want and use a single elasticsearch output with

index => "%{[@metadata][index_prefix]}-%{+YYYY}"

in the configuration. Perhaps this would work:

if "system" not in [tags] and "nginx" not in [tags] {
  mutate {
    replace => {
      "[@metadata][index_prefix]" => "other-%{env}"
    }
  }
}
1 Like
5

Hi @magnusbaeck,

It doesn't work well.

It gets the index name like this:

"%{agent}-%{env}-2018.02"
6

Here is my complete logstash pipline configuration ：

both system and nginx tags are working well for generating the index what I want .
for neither system nor nginx tags, the variables can't be substituted. %{agent}-%{env}-2018.02


#logstash for tlsbeat
input {
  tcp {
    port => 5045
  }
}


filter {

  if "_jsonparsefailure" in [tags] {
        drop { }
  }
  
  json {
    source => "message"
  }
  
  mutate {
    remove_field =>  ["message"]
    add_field => { "[@metadata][index_prefix]" => "%{agent}-%{env}-%{service}" }  
    remove_field =>  ["agent","env","host","port","service"]
  }
  
}

output {
  
  if "system" in [tags] {  
    elasticsearch {
      hosts => ["esdatatest01.tls.ad:9200","esdatatest02.tls.ad:9200","esdatatest03.tls.ad:9200","esdatatest04.tls.ad:9200","esdatatest05.tls.ad:9200","esdatatest06.tls.ad:9200","esdatatest07.tls.ad:9200",
               "esdatatest08.tls.ad:9200","esdatatest09.tls.ad:9200","esdatatest10.tls.ad:9200","esdatatest11.tls.ad:9200","esdatatest12.tls.ad:9200","esdatatest13.tls.ad:9200","esdatatest14.tls.ad:9200",
               "esdatatest15.tls.ad:9200","esdatatest16.tls.ad:9200","esdatatest17.tls.ad:9200","esdatatest18.tls.ad:9200","esdatatest19.tls.ad:9200","esdatatest20.tls.ad:9200"]
      template_overwrite => false
      manage_template => false
      index => "%{[@metadata][index_prefix]}-%{+YYYY.MM}"
      sniffing => false

    }      
  } 
  
  if "nginx" in [tags] {  
    elasticsearch {
      hosts => ["esdatatest01.tls.ad:9200","esdatatest02.tls.ad:9200","esdatatest03.tls.ad:9200","esdatatest04.tls.ad:9200","esdatatest05.tls.ad:9200","esdatatest06.tls.ad:9200","esdatatest07.tls.ad:9200",
               "esdatatest08.tls.ad:9200","esdatatest09.tls.ad:9200","esdatatest10.tls.ad:9200","esdatatest11.tls.ad:9200","esdatatest12.tls.ad:9200","esdatatest13.tls.ad:9200","esdatatest14.tls.ad:9200",
               "esdatatest15.tls.ad:9200","esdatatest16.tls.ad:9200","esdatatest17.tls.ad:9200","esdatatest18.tls.ad:9200","esdatatest19.tls.ad:9200","esdatatest20.tls.ad:9200"]
      template_overwrite => false
      manage_template => false
      index => "%{[@metadata][index_prefix]}-%{+YYYY}"
      sniffing => false
    }      
  }
  
  if "system" not in [tags] and "nginx" not in [tags] {
    elasticsearch {
      hosts => ["esdatatest01.tls.ad:9200","esdatatest02.tls.ad:9200","esdatatest03.tls.ad:9200","esdatatest04.tls.ad:9200","esdatatest05.tls.ad:9200","esdatatest06.tls.ad:9200","esdatatest07.tls.ad:9200",
               "esdatatest08.tls.ad:9200","esdatatest09.tls.ad:9200","esdatatest10.tls.ad:9200","esdatatest11.tls.ad:9200","esdatatest12.tls.ad:9200","esdatatest13.tls.ad:9200","esdatatest14.tls.ad:9200",
               "esdatatest15.tls.ad:9200","esdatatest16.tls.ad:9200","esdatatest17.tls.ad:9200","esdatatest18.tls.ad:9200","esdatatest19.tls.ad:9200","esdatatest20.tls.ad:9200"]
      template_overwrite => false
      manage_template => false
      index => "%{agent}-%{env}-%{+YYYY.MM}"
      sniffing => false
    }    
  } 
}
7

please help me out,
I need to get it done as soon as possible, cuz Chinese new year is coming soon.

8

Something along these lines should work:

filter {
  if "system" in [tags] { 
    mutate {
      add_field => { "[@metadata][index_prefix]" => "%{agent}-%{env}-%{service}" }
    }
  } else if "nginx" in [tags] {
    mutate {
      add_field => { "[@metadata][index_prefix]" => "%{agent}-%{env}-%{service}" }
    }
  } else {
    mutate {
      add_field => { "[@metadata][index_prefix]" => "%{agent}-%{env}" }
    }
  }

  mutate {
    remove_field =>  ["agent","env","host","port","service","message"]
  }
}

output { 
  elasticsearch {
    hosts => ["esdatatest01.tls.ad:9200","esdatatest02.tls.ad:9200","esdatatest03.tls.ad:9200","esdatatest04.tls.ad:9200","esdatatest05.tls.ad:9200","esdatatest06.tls.ad:9200","esdatatest07.tls.ad:9200",
               "esdatatest08.tls.ad:9200","esdatatest09.tls.ad:9200","esdatatest10.tls.ad:9200","esdatatest11.tls.ad:9200","esdatatest12.tls.ad:9200","esdatatest13.tls.ad:9200","esdatatest14.tls.ad:9200",
               "esdatatest15.tls.ad:9200","esdatatest16.tls.ad:9200","esdatatest17.tls.ad:9200","esdatatest18.tls.ad:9200","esdatatest19.tls.ad:9200","esdatatest20.tls.ad:9200"]
    template_overwrite => false
    manage_template => false
    index => "%{[@metadata][index_prefix]}-%{+YYYY.MM}"
    sniffing => false
  }      
}
3 Likes
9

It gets the index name like this:

"%{agent}-%{env}-2018.02"

Of course, because you're deleting those fields in your mutate filter.

2 Likes
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.