Cant's set log_time to @timestamp

env:

logstash 6.2
ES: 6.2

I touch two config files in /etc/logstash/conf.d , eg:a.conf b.conf

a.conf input from kafka, setting type is 'aa'

b.conf input from file, setting type is 'bb'

Both output are same ES

Question:
aa.conf can set log_time to @timestamp ,but bb.conf can't set log_time to @timestamp

aa.conf log time format:

2018-08-16 17:45:42.299

aa.conf

input {
  kafka {
        bootstrap_servers => "kafka0:19092,kafka1:19093,kafka2:19094"
        topics => [ "aa" ]
        codec => "json"
        type => "aa"
        group_id => "aa"
        consumer_threads => 2

  }
}

filter {
  if [type] == "aa" {
    grok {
          match => [ "message" , "(20%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{TIME:time})\s+%{LOGLEVEL:level}" ]
    }
  
    mutate {
         add_field => [ "log_time","20%{year}-%{month}-%{day} %{time}" ]
    }
  
    date {
          match => [ "log_time","yyyy-MM-dd HH:mm:ss.SSS" ]
          target => "@timestamp"
    }
  
    ruby {
          code => [ "event.set('index_day', event.get('@timestamp').time.localtime.strftime('%Y.%m.%d'))" ]
    }
  
    mutate {
         remove_field => ["[beat][name]","[beat][version]","@version","offset","tmptime","log_time","year","month","day","time"]
    }
  }
}

output {
  if [type] == "aa" {
      elasticsearch {
        codec => plain{ charset => "UTF-8" }
        hosts => "http://es1:9200"
        index => "%{[fields][log_topic]}-%{index_day}"
      }
  }
}

bb.conf log time format

[23:59:53.025]2018-08-14 23:59:53

bb.conf

input {
  file {
      path => [ "/data/bba/mobile/*" ]
      start_position => "beginning"
      type => "bb"
      codec=> multiline {
         pattern => "^\[([0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3})\]([0-9]{4}-[0-9]{2}-[0-9]{2})"
         negate => true
         what => "previous"
      }
  }
}



filter {
  if [type] == "bb" {
     grok {
           match => [ "message" , "\[%{TIME:time}\](20%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{HOUR}:%{MINUTE}:%{SECOND}) %{LOGLEVEL}" ]
     }

     mutate {
          add_field => [ "log_time","20%{year}-%{month}-%{day} %{time}" ]
     }

     date {
           match => [ "log_time","yyyy-MM-dd HH:mm:ss.SSS" ]
           target => "@timestamp"
     }
  }
}

output {

  if [type] == "bb" {
    elasticsearch {
      codec => plain{ charset => "UTF-8" }
      hosts => "http://es1:9200"
      index => "bb"
    }
  }

}

How can I debug logstash application configfile? I just only see the result in ES now!

I had tried set "level: debug" in logstash.yml,but it have too many logs

Please show an example bb event produced by Logstash. Copy/paste the raw JSON document from Kibana's JSON tab.

Thank you for reply!

I am sorry ! It is my fault! I make a worng "type"! It is working now!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.