Hi,
I have two different patterns in a single log file and I have two different groks handling them. But, somehow, the second grok is not able to parse the logs. I checked in grok debugger and everything is parsed correctly.
Here's the logstash config:
if "devops-logs" in [tags] {
grok {
patterns_dir => ["/etc/logstash/conf.d/patterns"]
match => { "message" => "%{DATE_CUSTOM:date}%{SPACE}%{TIME:time}%{SPACE}%{DATA}%{SPACE}%{LOGLEVEL:log_level}%{SPACE}%{SPACE}%{JOB_NAME}%{SPACE}-%{SPACE}tag%{SPACE}:%{SPACE}%{TAG_NAME:tag},%{SPACE}instance%{SPACE}:%{SPACE}%{NUMBER:instance},%{SPACE}event_date%{SPACE}:%{SPACE}%{DATE_CUSTOM:event_date}%{DATA},%{SPACE}count%{SPACE}:%{SPACE}%{NUMBER:count},%{SPACE}quota_amount%{SPACE}:%{SPACE}%{QUOTA_AMOUNT:quota_amount}" }
tag_on_failure => ["no-overage-per-quota"]
}
grok {
patterns_dir => ["/etc/logstash/conf.d/patterns"]
match => { "message" => "%{DATE_CUSTOM:date}%{SPACE}%{TIME:time}%{SPACE}%{DATA}%{SPACE}%{LOGLEVEL:log_level}%{SPACE}%{SPACE}%{JOB_NAME}%{SPACE}-%{SPACE}tag%{SPACE}:%{SPACE}%{TAG_NAME:tag},%{SPACE}instance%{SPACE}:%{SPACE}%{NUMBER:instance},%{SPACE}event_date%{SPACE}:%{SPACE}%{DATE_CUSTOM:event_date}%{DATA},%{SPACE}count%{SPACE}:%{SPACE}%{NUMBER:count},%{SPACE}%{DATA}%{SPACE}:%{SPACE}%{NUMBER:event_id}" }
remove_tag => ["no-overage-per-quota"]
}
mutate {
convert => { "event_id" => "integer" }
convert => { "count" => "integer" }
convert => { "instance" => "integer" }
convert => { "tag" => "string" }
convert => { "quota_amount" => "string" }
}
}
Here's the sample log entry which is not getting parsed:
2018-05-21 05:00:01.279 [pool-3-thread-1] INFO a.b.c.devops.SqlQueryOutputLoggerJob:34 - tag : events-per-quota, instance : 433974071000110045, event_date : 2018-05-21 00:00:00.0, count : 2142, event_id : 11