Formatting data as a block in logstash

Below is my sample data. How can it be filter based on start time and end time using grok pattern. Any idea friends since the data came as a stream of blocks. Can you guys suggest and provid example. I need to know start time , end time, success or fail. if fail the messge

++++++++++++++++++++++++++++++++++++++++++++++
Name: My file name
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Begin Date:	MM/DD/YYYY
Begin Time:	HH:MM:SS

Activity
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
YYYY-MM-DD HH:MM:SS message
YYYY-MM-DD HH:MM:SS message
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Result
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
messge Completed
End Date:	MM/DD/YYYY
End Time:	HH:MM:SS
Executed by: Name
++++++++++++++++++++++++++++++++++++++++++++++

++++++++++++++++++++++++++++++++++++++++++++++
Name: My file name
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Begin Date:	MM/DD/YYYY
Begin Time:	HH:MM:SS

Activity
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
YYYY-MM-DD HH:MM:SS message
YYYY-MM-DD HH:MM:SS message
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Result
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
message Failed
ERROR:	message
ERROR:	message
  at stacktrace
  at stacktrace
Caused by: message
  at stacktrace
  at stacktrace
	
End Date:	MM/DD/YYYY
End Time:	HH:MM:SS
Executed by: Name
++++++++++++++++++++++++++++++++++++++++++++++

A couple of hints I can think of.
First you need to use a multiline input and capture everything between the "++++++" lines.
Like this:
++++++++++++++++++++++++++++++++++++++++++++++ Name: My file name ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Begin Date: 10/13/1987 Begin Time: HH:MM:SS Activity ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ YYYY-MM-DD HH:MM:SS message YYYY-MM-DD HH:MM:SS message ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Result ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ messge Completed End Date: MM/DD/YYYY End Time: HH:MM:SS Executed by: Name ++++++++++++++++++++++++++++++++++++++++++++++

Then use a grok pattern (going to be ugly) to cut out the pieces that you need.

I started playing around with it a little bit (assuming that all newline characters were removed and the entire thing could be handled as a single line event.

\+ Name: %{DATA:my_file_name} \~%{NOTSPACE} Begin Date: %{NUMBER:Month}

This will get you this:


  "my_file_name": [
    [
      "My file name"
    ]
  ],
  "NOTSPACE": [
    [
      "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
    ]
  ],
  "Month": [
    [
      "10"
    ]
  ],
  "BASE10NUM": [
    [
      "10"
    ]
  ]
}

Check out https://grokdebug.herokuapp.com/ to help you figure out the rest.
Good luck.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.