Could not split in multiline Logs in Logstash

Elastic Stack Logstash
1

Greetings,
I am trying to create filter for my log file.
My log file is :

=-=-=-=-=-=-=-=-
Timestamp: Thursday, April 19, 2018 2:48:54 AM
Message: ID: 7
An exception of type 'System.Exception' occurred and was caught.
=-=-=-=-=-=-=-=-
=-=-=-=-=-=-=-=-
Timestamp: Thursday, April 19, 2018 2:48:54 AM
Message: ID: 8
An exception of type 'System.Exception' occurred and was caught.
=-=-=-=-=-=-=-=-

=-=-=-=-=-=-=-=- denote start and end of my log file.
My config file is :

input {
beats {
        port => "5044"
    }
}
filter {
multiline {
	   pattern => "^=-=-=-=-=-=-=-=-"
	   negate => true
	   what => previous
    }
grok {
       match => [ "message", "Timestamp:\s%{DAY},\s%{MONTH}\s%{MONTHDAY},\s%{YEAR}\s%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?" ]
       add_field => {
	        "LogTimestamp" => "Timestamp:\s%{DAY},\s%{MONTH}\s%{MONTHDAY},\s%{YEAR}\s%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?"
	   }
    }	
}
output {
elasticsearch {
        hosts => [ "localhost:9200" ]
	}
}

This is creating 2 documents(rows) for each log. One containing separator+log and other contains separator only.
(A) I want to delete the document containing only separator.
(B) I want to split the message field by creating new field 'LogTimestamp' that contain my Log timestamp and 'message' field contain rest of the message.
Like this: LogTimestamp: Thursday, April 19, 2018 2:48:54 AM Message: Message: ID: 7 An exception of type 'System.Exception' occurred and was caught.
I am beginner and new to grok patterns and filters. Please help,i am struck here from past 2 days. Also let me know if there is any proper documentation to learn about grok filters from.

2

Do not use the multiline filter in Logstash as this has been deprecated. You should always perform multiline processing as close to the source as possible, which in this case is in Filebeat.

3

@Christian_Dahlqvist Okay, i will apply it in filebeat. Meanwhile , can you tell me where i am wrong. I am trying to create a new field out of "message" field. My config file is :
# The # character at the beginning of a line indicates a comment. Use
# comments to describe your configuration.
input {
beats {
port => "5044"
}
}
# The filter part of this file is commented out to indicate that it is
# optional.
filter {
grok {
match => { "message" => "%{DATA:timestamp}"}
}
mutate {
add_field => {
"LogTimestamp" => "%{timestamp}"
}
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
}
}

4

DATA will match the entire message, so this grok pattern does not make any sense to me. Have a look at this blog post for an introduction on how to work with Logstash and parse data.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.