env：
logstash 6.2
ES: 6.2
I touch two config files in /etc/logstash/conf.d , eg：a.conf b.conf
a.conf input from kafka, setting type is 'aa'
b.conf input from file, setting type is 'bb'
Both output are same ES
Question:
aa.conf can set log_time to @timestamp ,but bb.conf can't set log_time to @timestamp
aa.conf log time format:
2018-08-16 17:45:42.299
aa.conf
input {
kafka {
bootstrap_servers => "kafka0:19092,kafka1:19093,kafka2:19094"
topics => [ "aa" ]
codec => "json"
type => "aa"
group_id => "aa"
consumer_threads => 2
}
}
filter {
if [type] == "aa" {
grok {
match => [ "message" , "(20%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{TIME:time})\s+%{LOGLEVEL:level}" ]
}
mutate {
add_field => [ "log_time","20%{year}-%{month}-%{day} %{time}" ]
}
date {
match => [ "log_time","yyyy-MM-dd HH:mm:ss.SSS" ]
target => "@timestamp"
}
ruby {
code => [ "event.set('index_day', event.get('@timestamp').time.localtime.strftime('%Y.%m.%d'))" ]
}
mutate {
remove_field => ["[beat][name]","[beat][version]","@version","offset","tmptime","log_time","year","month","day","time"]
}
}
}
output {
if [type] == "aa" {
elasticsearch {
codec => plain{ charset => "UTF-8" }
hosts => "http://es1:9200"
index => "%{[fields][log_topic]}-%{index_day}"
}
}
}
bb.conf log time format
[23:59:53.025]2018-08-14 23:59:53
bb.conf
input {
file {
path => [ "/data/bba/mobile/*" ]
start_position => "beginning"
type => "bb"
codec=> multiline {
pattern => "^\[([0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3})\]([0-9]{4}-[0-9]{2}-[0-9]{2})"
negate => true
what => "previous"
}
}
}
filter {
if [type] == "bb" {
grok {
match => [ "message" , "\[%{TIME:time}\](20%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{HOUR}:%{MINUTE}:%{SECOND}) %{LOGLEVEL}" ]
}
mutate {
add_field => [ "log_time","20%{year}-%{month}-%{day} %{time}" ]
}
date {
match => [ "log_time","yyyy-MM-dd HH:mm:ss.SSS" ]
target => "@timestamp"
}
}
}
output {
if [type] == "bb" {
elasticsearch {
codec => plain{ charset => "UTF-8" }
hosts => "http://es1:9200"
index => "bb"
}
}
}
How can I debug logstash application configfile? I just only see the result in ES now!
I had tried set "level: debug" in logstash.yml,but it have too many logs