Possible bug in logstash 7.16.2 with degug

Elastic Stack Logstash
1

Hi,
We recently upgraded our stack to elastic 7.16.2, and after that I introduced some changes in my configuracion. That configuration worked in test, but when i put it in production it does not get any error, however it does not index data. As in production i dont have debug enabled, i decided to turn it on, and when i turn it on it index information, i turn it off and not information indexed.

My configuration consists of several pipelines

main -> pipe1 --------┐
        Pipe2 ---------┐
        Pipe3 --------------------->  es
                     └------------->  fs
        ...... ------------┘
        PipeN -------┘

One main pipeline with several input, then depending on data its forwarded to individual pipelines, and all pipelines after procesing send to es (that writes to Elasticsearch) and to fs (writes to filesystem)

This is the configuration of es pipeline that if debug enabled indexes, and if not no:

input { pipeline { address => "es" } }

filter {
  if (!([tags]) or ([tags] and !([tags][0]))) and [meta][pipeline] != "discarded" {
     mutate { add_field => { "[@metadata][pipeline]" => "%{[meta][pipeline]}" } }
#     mutate { add_field => { "[@metadata][fecha_part]" => "%{+YYYY.MM}" } }
     if [meta][pipeline] == "okd-app" {
         mutate { add_field => { "[@metadata][fecha_part]" => "%{+xxxx.ww}" } }
     } else {
         mutate { add_field => { "[@metadata][fecha_part]" => "%{+YYYY.MM}" } }
     }
  } else {
     mutate { add_field => { "[@metadata][pipeline]" => "discarded-%{[meta][input]}" } }
     mutate { add_field => { "[@metadata][fecha_part]" => "%{+YYYY.MM.dd}" } }
  }

}

output {
  elasticsearch {
    hosts => ["xxx",  "xxx",  "xxx"]
    cacert => '/etc/logstash/certs/rootCA.cer'
    ssl => true
    keystore => "/etc/logstash/certs/my_keystore.pfx"
    keystore_password => ""
    user => '${es_usr}'
    password => '${es_pwd}'
    index => "%{[@metadata][pipeline]}-xxx-%{[@metadata][fecha_part]}"
  }
}

if i put the the following, and debug disabled it indexes

input { pipeline { address => "es" } }

filter {
  if (!([tags]) or ([tags] and !([tags][0]))) and [meta][pipeline] != "discarded" {
     mutate { add_field => { "[@metadata][pipeline]" => "%{[meta][pipeline]}" } }
     mutate { add_field => { "[@metadata][fecha_part]" => "%{+YYYY.MM}" } }
  } else {
     mutate { add_field => { "[@metadata][pipeline]" => "discarded-%{[meta][input]}" } }
     mutate { add_field => { "[@metadata][fecha_part]" => "%{+YYYY.MM.dd}" } }
  }

}

output {
  elasticsearch {
    hosts => ["xxx",  "xxx",  "xxx"]
    cacert => '/etc/logstash/certs/rootCA.cer'
    ssl => true
    keystore => "/etc/logstash/certs/my_keystore.pfx"
    keystore_password => ""
    user => '${es_usr}'
    password => '${es_pwd}'
    index => "%{[@metadata][pipeline]}-xxx-%{[@metadata][fecha_part]}"
  }
}

Dont know if theres any problem with the date format or what.

Where can I report it?
thanks

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.