Maybe. You would have to try it. There is a comment in the code that "We are making an assumption concerning cloudfront log format, the user will use the plain or the line codec", which suggests you cannot, but also a comment "ensure any stateful codecs (such as multi-line ) are flushed" which suggests you can.
Sir, we've about more than 10-50K+ files on S3 buckets, some logs patterns require multiline codec to get single logs lines before applying filter. For that purpose I would need to first segregate the multilines and single lines logs in S3 that is nearly impossible for me at this time. Is there a way I get all logs of different type and use something in filter to segregate one from other.
Like if [message] =~ /this pattern/ {do this} else if [message] =~ /next pattern/ {do that.. }
If you need to make the use a codec conditional on the contents you could use a tcp input and output to connect two pipelines and conditionally send the events to the second pipeline, which could use a multiline codec.
You would need event order to be preserved, so you have to set pipeline.workers to 1 and possibly set pipeline.ordered.
Here my ambiguity is about multiline.yml file. I'm adding two things in input, 1st is the input listener and 2nd one is the codec. Don't know it will work or not.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.