Logstash - Parsing Log Messages

Elastic Stack Logstash
1

Hi,

I have a log file which contains a timestamp and logs message as shown below:
I need one multiple pattern such that if the line does not start with the timestamp it should be joined with the preceding line. Can anyone post me how input plugin should be written for this scenario.

Here is my log message:

10-05-2018 00:00:00.0031 DEBUG Thd: 6540 [Schedule] (checkSchedules) Chequea a: 10/05/2018 12:00:00 AM ON 10/05/2018 12:00:00 AM
10-05-2018 00:00:00.0031 DEBUG Thd: 6540 [Schedule] (checkSchedules) ThreadFinished: 00:00:00
10-05-2018 00:00:01.1904 ERROR Thd: 6568 192.168.56.1 [ManagedObject] (EntityLoad) Object not be loaded 192.168.56.1 AGENT
Adecef.TelgatNGOSS.ObjectServer.ManagedObjectException: instance does not exists
at Adecef.TelgatNGOSS.ObjectServer.ManagedObject.EntityLoad()
10-05-2018 00:00:01.1904 ERROR Thd: 6568 [ObjectServer] (GetManagedObject) Object can not be load: 192.168.56.1
Adecef.TelgatNGOSS.ObjectServer.ManagedObjectException: instance does not exists
at Adecef.TelgatNGOSS.ObjectServer.ManagedObject.EntityLoad()
at Adecef.TelgatNGOSS.ObjectServer.MAgentObject.EntityLoad()
at Adecef.TelgatNGOSS.ObjectServer.ObjectFactory.GetManagedObject(String oid, String objectClass, Boolean isFromWeb)
10-05-2018 00:00:01.2373 ERROR Thd: 6568 [ObjectServer] (processEvent) Exception Object reference not set to an instance of an object.
System.NullReferenceException: Object reference not set to an instance of an object.
at Adecef.TelgatNGOSS.ObjectServer.EventReportManager.processEvent(Object taskInfo)
10-05-2018 00:00:08.6432 ERROR Thd: 6568 192.168.56.1 [ManagedObject] (EntityLoad) Object not be loaded 192.168.56.1 AGENT
Adecef.TelgatNGOSS.ObjectServer.ManagedObjectException: instance does not exists
at Adecef.TelgatNGOSS.ObjectServer.ManagedObject.EntityLoad()
10-05-2018 00:00:08.6432 ERROR Thd: 6568 [ObjectServer] (GetManagedObject) Object can not be load: 192.168.56.1
Adecef.TelgatNGOSS.ObjectServer.ManagedObjectException: instance does not exists
at Adecef.TelgatNGOSS.ObjectServer.ManagedObject.EntityLoad()
at Adecef.TelgatNGOSS.ObjectServer.MAgentObject.EntityLoad()
at Adecef.TelgatNGOSS.ObjectServer.ObjectFactory.GetManagedObject(String oid, String objectClass, Boolean isFromWeb)
10-05-2018 00:00:09.5814 ERROR Thd: 6568 [ObjectServer] (processEvent) Exception Object reference not set to an instance of an object.
System.NullReferenceException: Object reference not set to an instance of an object.
at Adecef.TelgatNGOSS.ObjectServer.EventReportManager.processEvent(Object taskInfo)
10-05-2018 00:00:11.2221 ERROR Thd: 7156 192.168.56.1 [ManagedObject] (EntityLoad) Object not be loaded 192.168.56.1 AGENT
Adecef.TelgatNGOSS.ObjectServer.ManagedObjectException: instance does not exists
at Adecef.TelgatNGOSS.ObjectServer.ManagedObject.EntityLoad()

2

The Logstash documentation contains an example of something very similar: https://www.elastic.co/guide/en/logstash/current/multiline.html

3

I have used multiline codec, but its not working as expected, I mean it is taking all lines between two lines, only it is taking 2 or 3 lines. And remaining lines are going with next event.

This is configuration file:

input {
file {
path => ["D:/Sample/*"]
start_position => "beginning"
sincedb_path => "/dev/null"
codec => multiline {
pattern => "^(\s)"
what => "previous"
}
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
index => "sample-%{+YYYY.MM.dd}"
}
stdout {}

}

4

Please show an example of the input and the resulting events. Use a stdout { codec => rubydebug } output to dump the raw events and make sure you format the example input as preformatted text.

5

@magnusbaeck

The Contents of my log file:

2018-05-10 00:00:00.0031 DEBUG Thd: 6540 [Schedule] (checkSchedules) Chequea a: 10/05/2018 12:00:00 AM ON
2018-05-10 00:00:00.0031 DEBUG Thd: 6540 [Schedule] (checkSchedules) ThreadFinished: 00:00:00
2018-05-10 00:00:01.1904 ERROR Thd: 6568 192.168.56.1 [ManagedObject] (EntityLoad) Object not be loaded 192.168.56.1 AGENT
Adecef.TelgatNGOSS.ObjectServer.ManagedObjectException: instance does not exists
at Adecef.TelgatNGOSS.ObjectServer.ManagedObject.EntityLoad()
2018-05-10 00:00:01.1904 ERROR Thd: 6568 [ObjectServer] (GetManagedObject) Object can not be load: 192.168.56.1
Adecef.TelgatNGOSS.ObjectServer.ManagedObjectException: instance does not exists
at Adecef.TelgatNGOSS.ObjectServer.ManagedObject.EntityLoad()
at Adecef.TelgatNGOSS.ObjectServer.MAgentObject.EntityLoad()
at Adecef.TelgatNGOSS.ObjectServer.ObjectFactory.GetManagedObject(String oid, String objectClass, Boolean isFromWeb)

This is my Input

input {
file {
path => ["D:/TestLogs/tm*"]
start_position => "beginning"
sincedb_path => "/dev/null"
codec => multiline {
pattern => "^(\s)"
what => "previous"
}
}
}

and the output got generated is shown below:

{
"@timestamp" => 2018-05-23T04:56:10.448Z,
"path" => "D:/TestLogs/tm20180510",
"host" => "BALP-SunilS",
"@version" => "1",
"message" => "2018-05-10 00:00:01.1904\tERROR\tThd: 6568\t\t[ObjectServer]\t(GetManagedObject)\tObject can not be load: 192.168.56.1\r"
}
{
"path" => "D:/TestLogs/tm20180510",
"host" => "BALP-SunilS",
"tags" => [
[0] "multiline"
],
"message" => "Adecef.TelgatNGOSS.ObjectServer.ManagedObjectException: instance does not exists\r\n at Adecef.TelgatNGOSS.ObjectServer.ManagedObject.EntityLoad()\r",
"@timestamp" => 2018-05-23T04:56:10.448Z,
"@version" => "1"
}

6

I repeat: Make sure you format the example input as preformatted text. Use Markdown notation or the </> toolbar button.

But I can see in your output that the second line of the message doesn't begin with a whitespace character, it starts with "Adecef". That's why the multiline codec doesn't work. Have a look at this example instead: https://www.elastic.co/guide/en/logstash/current/multiline.html#_timestamps

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.