Multiple Line in Single Event

Elastic Stack Logstash
1

Hi,

Consider the below example:

Logfile:

START
INFO: NOTIFICATION_ID,NOTIFICATION_TYPE,NOTIFICATION_DATE,ORDER_REFERENCE,US_ID,SIO_ID,ORIGINAL_ORDER_REFERENCE,ACCESS_QUOTE_REF,PHONE,ADDRESS1,ADDRESS2,ADDRESS3,POSTAL_CODE,CITY,STATE,COUNTRY
Jun 05, 2017 6:45:07 AM sai.pollers.SaiPollerServlet info
END

The above logfile contains the two delimeter START and END.
Now I want to print the log between these two delimeter in Single Event every time.

Can you please share the configuration required for this in logstash.

Regards,
Anupam

2

Can anyone help me for my query.

Regards,
Anupam

3

Do all log entries in this log file start with START and end with END, or are there other types of entries? If so, what do these look like?

4

These delimeter (START and END) will be unique.

5

If all events have the same structure, you should be able use the a multiline codec with your input to groups all lines following START (up until the next START) into a single event:

codec => multiline {
  pattern => "^START"
  negate => true
  what => previous
  auto_flush_interval => 1
}
6

Hi,

Thanks.

This will fail in two cases.

  1. If the logs is in below format then it won't work.

START // Start of Log
INFO: NOTIFICATION_ID,NOTIFICATION_TYPE,NOTIFICATION_DATE,ORDER_REFERENCE,US_ID,SIO_ID,ORIGINAL_ORDER_REFERENCE,ACCESS_QUOTE_REF,PHONE,ADDRESS1,ADDRESS2,ADDRESS3,POSTAL_CODE,CITY,STATE,COUNTRY
Jun 05, 2017 6:45:07 AM sai.pollers.SaiPollerServlet info
END // End of Log
83.149.9.216 - - [04/Jan/2015:05:13:44 +0000] "GET /presentations/logstash-monitorama-2013/plugin/highlight/highlight.js HTTP/1.1" 200 26185 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0
83.149.9.216 - - [04/Jan/2015:05:13:44 +0000] "GET /presentations/logstash-monitorama-2013/plugin/highlight/highlight.js HTTP/1.1" 200 26185 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0
START // Start of Log
83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/
END // End of Log

I doesn't need the line between END to START i.e; between end delimeter and start delimeter.

2)The name of delimeter is in my control. I can also do the same for both start as START and end as START. But this will also fail.

START // Start of Log
INFO: NOTIFICATION_ID,NOTIFICATION_TYPE,NOTIFICATION_DATE,ORDER_REFERENCE,US_ID,SIO_ID,ORIGINAL_ORDER_REFERENCE,ACCESS_QUOTE_REF,PHONE,ADDRESS1,ADDRESS2,ADDRESS3,POSTAL_CODE,CITY,STATE,COUNTRY
Jun 05, 2017 6:45:07 AM sai.pollers.SaiPollerServlet info
START // End of Log
83.149.9.216 - - [04/Jan/2015:05:13:44 +0000] "GET /presentations/logstash-monitorama-2013/plugin/highlight/highlight.js HTTP/1.1" 200 26185 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0
83.149.9.216 - - [04/Jan/2015:05:13:44 +0000] "GET /presentations/logstash-monitorama-2013/plugin/highlight/highlight.js HTTP/1.1" 200 26185 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0
START // Start of Log
83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/
START // End of Log

Can you please suggest so that I can get the logs between start and end delimeters.

Thanks,
Anupam

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.