Logstash - Merge multiple lines into one event

Elastic Stack Logstash
1

Hi,

I want to create one configuration file for logstash such that it should generate an event and upload a whole message whatever is there in one timestamp to other timestamp.
Can anyone please help me how to achieve this.

Here is my log as shown below:

10-05-2018 00:00:00.0031 DEBUG Thd: 6540 [Schedule] (checkSchedules) Chequea a: 10/05/2018 12:00:00 AM ON 10/05/2018 12:00:00 AM
10-05-2018 00:00:00.0031 DEBUG Thd: 6540 [Schedule] (checkSchedules) ThreadFinished: 00:00:00
10-05-2018 00:00:01.1904 ERROR Thd: 6568 192.168.56.1 [ManagedObject] (EntityLoad) Object not be loaded 192.168.56.1 AGENT
Adecef.TelgatNGOSS.ObjectServer.ManagedObjectException: instance does not exists
at Adecef.TelgatNGOSS.ObjectServer.ManagedObject.EntityLoad()
10-05-2018 00:00:01.1904 ERROR Thd: 6568 [ObjectServer] (GetManagedObject) Object can not be load: 192.168.56.1
Adecef.TelgatNGOSS.ObjectServer.ManagedObjectException: instance does not exists
at Adecef.TelgatNGOSS.ObjectServer.ManagedObject.EntityLoad()
at Adecef.TelgatNGOSS.ObjectServer.MAgentObject.EntityLoad()
at Adecef.TelgatNGOSS.ObjectServer.ObjectFactory.GetManagedObject(String oid, String objectClass, Boolean isFromWeb)
10-05-2018 00:00:01.2373 ERROR Thd: 6568 [ObjectServer] (processEvent) Exception Object reference not set to an instance of an object.
System.NullReferenceException: Object reference not set to an instance of an object.
at Adecef.TelgatNGOSS.ObjectServer.EventReportManager.processEvent(Object taskInfo)
10-05-2018 00:00:08.6432 ERROR Thd: 6568 192.168.56.1 [ManagedObject] (EntityLoad) Object not be loaded 192.168.56.1 AGENT
Adecef.TelgatNGOSS.ObjectServer.ManagedObjectException: instance does not exists
at Adecef.TelgatNGOSS.ObjectServer.ManagedObject.EntityLoad()
10-05-2018 00:00:08.6432 ERROR Thd: 6568 [ObjectServer] (GetManagedObject) Object can not be load: 192.168.56.1
Adecef.TelgatNGOSS.ObjectServer.ManagedObjectException: instance does not exists
at Adecef.TelgatNGOSS.ObjectServer.ManagedObject.EntityLoad()
at Adecef.TelgatNGOSS.ObjectServer.MAgentObject.EntityLoad()
at Adecef.TelgatNGOSS.ObjectServer.ObjectFactory.GetManagedObject(String oid, String objectClass, Boolean isFromWeb)
10-05-2018 00:00:09.5814 ERROR Thd: 6568 [ObjectServer] (processEvent) Exception Object reference not set to an instance of an object.
System.NullReferenceException: Object reference not set to an instance of an object.
at Adecef.TelgatNGOSS.ObjectServer.EventReportManager.processEvent(Object taskInfo)
10-05-2018 00:00:11.2221 ERROR Thd: 7156 192.168.56.1 [ManagedObject] (EntityLoad) Object not be loaded 192.168.56.1 AGENT
Adecef.TelgatNGOSS.ObjectServer.ManagedObjectException: instance does not exists
at Adecef.TelgatNGOSS.ObjectServer.ManagedObject.EntityLoad()
10-05-2018 00:00:11.2221 ERROR Thd: 7156 [ObjectServer] (GetManagedObject) Object can not be load: 192.168.56.1
Adecef.TelgatNGOSS.ObjectServer.ManagedObjectException: instance does not exists
at Adecef.TelgatNGOSS.ObjectServer.ManagedObject.EntityLoad()
at Adecef.TelgatNGOSS.ObjectServer.MAgentObject.EntityLoad()
at Adecef.TelgatNGOSS.ObjectServer.ObjectFactory.GetManagedObject(String oid, String objectClass, Boolean isFromWeb)
10-05-2018 00:00:11.2221 ERROR Thd: 7156 [ObjectServer] (processEvent) Exception Object reference not set to an instance of an object.
System.NullReferenceException: Object reference not set to an instance of an object.
at Adecef.TelgatNGOSS.ObjectServer.EventReportManager.processEvent(Object taskInfo)
10-05-2018 00:00:16.4403 ERROR Thd: 7156 192.168.56.1 [ManagedObject] (EntityLoad) Object not be loaded 192.168.56.1 AGENT
Adecef.TelgatNGOSS.ObjectServer.ManagedObjectException: instance does not exists
at Adecef.TelgatNGOSS.ObjectServer.ManagedObject.EntityLoad()
10-05-2018 00:00:16.4403 ERROR Thd: 7156 [ObjectServer] (GetManagedObject) Object can not be load: 192.168.56.1
Adecef.TelgatNGOSS.ObjectServer.ManagedObjectException: instance does not exists
at Adecef.TelgatNGOSS.ObjectServer.ManagedObject.EntityLoad()
at Adecef.TelgatNGOSS.ObjectServer.MAgentObject.EntityLoad()
at Adecef.TelgatNGOSS.ObjectServer.ObjectFactory.GetManagedObject(String oid, String objectClass, Boolean isFromWeb)
10-05-2018 00:00:16.4564 ERROR Thd: 7156 [ObjectServer] (processEvent) Exception Object reference not set to an instance of an object.
System.NullReferenceException: Object reference not set to an instance of an object.
at Adecef.TelgatNGOSS.ObjectServer.EventReportManager.processEvent(Object taskInfo)
10-05-2018 00:00:21.3002 ERROR Thd: 6568 192.168.56.1 [ManagedObject] (EntityLoad) Object not be loaded 192.168.56.1 AGENT
Adecef.TelgatNGOSS.ObjectServer.ManagedObjectException: instance does not exists
at Adecef.TelgatNGOSS.ObjectServer.ManagedObject.EntityLoad()
10-05-2018 00:00:21.3002 ERROR Thd: 6568 [ObjectServer] (GetManagedObject) Object can not be load: 192.168.56.1
Adecef.TelgatNGOSS.ObjectServer.ManagedObjectException: instance does not exists
at Adecef.TelgatNGOSS.ObjectServer.ManagedObject.EntityLoad()
at Adecef.TelgatNGOSS.ObjectServer.MAgentObject.EntityLoad()
at Adecef.TelgatNGOSS.ObjectServer.ObjectFactory.GetManagedObject(String oid, String objectClass, Boolean isFromWeb)
10-05-2018 00:00:21.3314 ERROR Thd: 6568 [ObjectServer] (processEvent) Exception Object reference not set to an instance of an object.
System.NullReferenceException: Object reference not set to an instance of an object.
at Adecef.TelgatNGOSS.ObjectServer.EventReportManager.processEvent(Object taskInfo)
10-05-2018 00:00:24.7846 INFO Thd: 5676 [Schedule] (mQueuedScheduleTimer_Elapsed) Chequea QueuedSchedules: 10/05/2018 12:00:24 AM
10-05-2018 00:00:24.8011 INFO Thd: 5676 [QueueCalls] (SelectPendingCalls) Queue Length: 0. Start checking resources by queue...
10-05-2018 00:00:24.8011 INFO Thd: 5676 [QueueCalls] (SelectPendingCalls) ...End checking resources by queue.
10-05-2018 00:00:26.2221 ERROR Thd: 6568 192.168.56.1 [ManagedObject] (EntityLoad) Object not be loaded 192.168.56.1 AGENT
Adecef.TelgatNGOSS.ObjectServer.ManagedObjectException: instance does not exists
at Adecef.TelgatNGOSS.ObjectServer.ManagedObject.EntityLoad()
10-05-2018 00:00:26.2221 ERROR Thd: 6568 [ObjectServer] (GetManagedObject) Object can not be load: 192.168.56.1
Adecef.TelgatNGOSS.ObjectServer.ManagedObjectException: instance does not exists
at Adecef.TelgatNGOSS.ObjectServer.ManagedObject.EntityLoad()
at Adecef.TelgatNGOSS.ObjectServer.MAgentObject.EntityLoad()
at Adecef.TelgatNGOSS.ObjectServer.ObjectFactory.GetManagedObject(String oid, String objectClass, Boolean isFromWeb)

2

Which lines do you want to combine?

3

Hi @Badger, @magnusbaeck

I want to combine lines which are there between two timestamp.

4

If you want to combine multiple lines of a stack trace then use a multiline codec. The documentation includes an example for a Java stack trace.

5

Hi @Badger

I have used multiline codec, but its not working as expected, I mean it is taking all lines between two lines, only it is taking 2 or 3 lines. And remaining lines are going with next event.

This is configuration file:

input {
file {
path => ["D:/Sample/*"]
start_position => "beginning"
sincedb_path => "/dev/null"
codec => multiline {
pattern => "^(\s)"
what => "previous"
}
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
index => "sample-%{+YYYY.MM.dd}"
}
stdout {}

}

6

The same as this. Please do not start multiple threads for exactly the same issue.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.