Here I am trying like use add_filed by catching when message contains startTime and create a new filed but I am not sure how I can append to next log because I believe when I close if condition it won't remember that filed to append to next log.
Unfortunately what you have is not a legal CEF-format log, but a log file that contains a mix of data in different formats, some lines of which are CEF, and others are a custom key/value format. This is going to make things difficult since no one codec is capable of processing all lines in a given log file.
This is made perhaps more difficult because Logstash (and likely your application that is doing the logging) does not make ordering guarantees, so there is no guarantee that a filter processing your events will see the start- and end-metadata events before receiving the related CEF log. It is also probable that your application will interleave messages from separate threads, which means you might get something like the following, where one start/end pair cannot be conclusively matched up with a specific CEF-line:
Emit the start and end timestamps within your CEF line, OR
Emit the start and end timestamps in CEF format AND include a unique "request identifier" or "activity identifier" in a CEF extension field so that the logs belonging to the same request or activity can be grouped together after the fact.
@yaauie That logs is originally from Incapsula website CEF log format example and I intentionally not added all metadata except start and End times, Also, I guarantee on logstash processing from top to bottom and for each logfile I have only one start and end times at the top.
All I need is if I process the logline of startTime , I want to hold the variable parsed from that line and injest and remaining CEF formated logs in my logfile
Logstash itself does not guarantee that a given filter will receive events that have been produced by your inputs in order; you can increase the liklihood that events are processed in order by reducing the number of pipeline workers to 1 and setting the batch size to 1 (which will significantly degrade performance), but even then we still are not guaranteed to process the lines in order 100% of the time.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.