Hi
I have configured logstash configuration to create an index in ES for all the logs present in three node cluster. Please find the below conf.
input {
file {
path => "D:\Development_Avecto\logstash-2.4.1\bin\dev_exceptions.txt"
start_position => "beginning"
sincedb_path => "NUL"
codec => multiline {
pattern => "^\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]"
negate => true
what => "previous"
}
}
}
filter {
grok {
match => [ "message", "(?m)^\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]\[%{LOGLEVEL:LEVEL}%{SPACE}\]\[%{DATA:ERRORTYPE}%{SPACE}\]%{SPACE}(?<ERRORMESSAGE>(.|\r|\n)*)"]
}
# DEBUG Logs
if "grokked" not in [tags] and "DEBUG" == [LEVEL] {
grok { match => [ "ERRORMESSAGE", "(?m)^\[%{DATA:SERVERNAME}\]" ]
add_tag => [ "Debug Logs", "grokked" ]
tag_on_failure => [ ]
}
}
# INFO Logs
if "grokked" not in [tags] and "INFO" == [LEVEL] {
grok { match => [ "ERRORMESSAGE", "(?m)^\[%{DATA:SERVERNAME}\]" ]
add_tag => [ "Info Logs", "grokked" ]
tag_on_failure => [ ]
}
}
# WARN Logs
if "grokked" not in [tags] and "WARN" == [LEVEL] {
grok { match => [ "ERRORMESSAGE", "(?m)^\[%{DATA:SERVERNAME}\]" ]
add_tag => [ "Warn Logs", "grokked" ]
tag_on_failure => [ ]
}
}
if "Exception" in [ERRORMESSAGE] {
mutate { add_field => { "Exception" => "Exception" }
}
}
date {
match => [ "TIMESTAMP" , "yyyy-MM-dd HH:mm:ss,SSS" ]
target => "TIMESTAMP"
}
mutate {
remove_field => ["message","@version","path","host","tags","type" ]
}
}
output {
elasticsearch { hosts => ["localhost:9200"]
index => "logstash-devlogs"
}
stdout { codec => rubydebug }
}
As per the conf file, a new field called "Exception"(specified in If condition) has created for the mentioned index.
And i need one more raw field should be created for the "Exception" field with respect to errors(for Eg: if it is a mapper parse exception or search parse exception or Invalid exception or Timeout exception)
Kindly post your comments ASAP. very urgent.