Below is my sample data. How can it be filter based on start time and end time using grok pattern. Any idea friends since the data came as a stream of blocks. Can you guys suggest and provid example. I need to know start time , end time, success or fail. if fail the messge
++++++++++++++++++++++++++++++++++++++++++++++
Name: My file name
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Begin Date: MM/DD/YYYY
Begin Time: HH:MM:SS
Activity
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
YYYY-MM-DD HH:MM:SS message
YYYY-MM-DD HH:MM:SS message
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Result
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
messge Completed
End Date: MM/DD/YYYY
End Time: HH:MM:SS
Executed by: Name
++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++
Name: My file name
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Begin Date: MM/DD/YYYY
Begin Time: HH:MM:SS
Activity
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
YYYY-MM-DD HH:MM:SS message
YYYY-MM-DD HH:MM:SS message
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Result
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
message Failed
ERROR: message
ERROR: message
at stacktrace
at stacktrace
Caused by: message
at stacktrace
at stacktrace
End Date: MM/DD/YYYY
End Time: HH:MM:SS
Executed by: Name
++++++++++++++++++++++++++++++++++++++++++++++
A couple of hints I can think of.
First you need to use a multiline input and capture everything between the "++++++" lines.
Like this: ++++++++++++++++++++++++++++++++++++++++++++++ Name: My file name ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Begin Date: 10/13/1987 Begin Time: HH:MM:SS Activity ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ YYYY-MM-DD HH:MM:SS message YYYY-MM-DD HH:MM:SS message ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Result ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ messge Completed End Date: MM/DD/YYYY End Time: HH:MM:SS Executed by: Name ++++++++++++++++++++++++++++++++++++++++++++++
Then use a grok pattern (going to be ugly) to cut out the pieces that you need.
I started playing around with it a little bit (assuming that all newline characters were removed and the entire thing could be handled as a single line event.
\+ Name: %{DATA:my_file_name} \~%{NOTSPACE} Begin Date: %{NUMBER:Month}
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.