# Multiline codec in filter **URL:** \ **Category:** Logstash\ **Tags:** elastic-stack-monitoring\ **Created:** [November 24, 2020, 2:41pm UTC](https://discuss.elastic.co/t/multiline-codec-in-filter/256520 "2020-11-24T14:41:59Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![shani](https://avatars.discourse-cdn.com/v4/letter/s/e95f7d/32.png) [@shani](https://discuss.elastic.co/u/shani)\ **Post date:** [November 24, 2020, 2:41pm UTC](https://discuss.elastic.co/t/multiline-codec-in-filter/256520/1 "2020-11-24T14:41:59Z")
Can I use a multiline codec in the filter? I've more than 10k+ files of 3 different patterns logs pattern on S3 bucket that will be fetched using input plugin. One log pattern requires a multiline codec before start parsing them. I want to use it like filter { if [message] =~ /this regex/ { - use multiline codec here ... - parse the logs } else if [message] =~ /and this regex/ { - do this and that } } I'm not able to use multiline codec in filter but when I use this in input, it works fine. Please guide me.. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 24, 2020, 3:01pm UTC](https://discuss.elastic.co/t/multiline-codec-in-filter/256520/2 "2020-11-24T15:01:39Z")
> [@shani](#): > > Can I use a multiline codec in the filter? No. codecs are called by inputs, not filters. ---
**Author:** ![shani](https://avatars.discourse-cdn.com/v4/letter/s/e95f7d/32.png) [@shani](https://discuss.elastic.co/u/shani)\ **Post date:** [November 24, 2020, 3:10pm UTC](https://discuss.elastic.co/t/multiline-codec-in-filter/256520/3 "2020-11-24T15:10:15Z")
Ok, but I've get files from S3 bucket. Can I use multiline codec on S3 input too or it is only specific for file input only? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 24, 2020, 3:14pm UTC](https://discuss.elastic.co/t/multiline-codec-in-filter/256520/4 "2020-11-24T15:14:23Z")
Maybe. You would have to try it. There is a comment in the [code](https://github.com/logstash-plugins/logstash-input-s3/blob/4e3f43a1edeeefd8a5d2b09c0aef3b1654c358fd/lib/logstash/inputs/s3.rb#L210) that "We are making an assumption concerning cloudfront log format, the user will use the plain or the line codec", which suggests you cannot, but also a comment "ensure any stateful codecs (such as multi-line ) are flushed" which suggests you can. ---
**Author:** ![shani](https://avatars.discourse-cdn.com/v4/letter/s/e95f7d/32.png) [@shani](https://discuss.elastic.co/u/shani)\ **Post date:** [November 24, 2020, 3:31pm UTC](https://discuss.elastic.co/t/multiline-codec-in-filter/256520/5 "2020-11-24T15:31:19Z")
Sir, we've about more than 10-50K+ files on S3 buckets, some logs patterns require multiline codec to get single logs lines before applying filter. For that purpose I would need to first segregate the multilines and single lines logs in S3 that is nearly impossible for me at this time. Is there a way I get all logs of different type and use something in filter to segregate one from other. Like if [message] =~ /this pattern/ {do this} else if [message] =~ /next pattern/ {do that.. } thank you ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 24, 2020, 4:17pm UTC](https://discuss.elastic.co/t/multiline-codec-in-filter/256520/6 "2020-11-24T16:17:52Z")
If you need to make the use a codec conditional on the contents you could use a tcp input and output to connect two pipelines and conditionally send the events to the second pipeline, which could use a multiline codec. You would need event order to be preserved, so you have to set pipeline.workers to 1 and possibly set pipeline.ordered. ---
**Author:** ![shani](https://avatars.discourse-cdn.com/v4/letter/s/e95f7d/32.png) [@shani](https://discuss.elastic.co/u/shani)\ **Post date:** [November 24, 2020, 7:09pm UTC](https://discuss.elastic.co/t/multiline-codec-in-filter/256520/7 "2020-11-24T19:09:14Z")
Sir, please correct me here: taking\_input.yml > input { fetching 50k+ logs files from S3 buckets} > filter {} > output { pipeline { send\_to =\> "parsing\_listener" } } parse.yml > input {pipeline { address =\> "parsing\_listener"} } > filter { > if [message] =~ /regex to detect single line data/ > { > mutate { add\_tag =\> ["single\_line\_data"] } > } > else if [message] =~ /regex to detect multiline data/ > { > mutate {add\_tag =\> ["multiline\_data"]} > } > } > output { > if "multiline\_data" in [tags] > { > send\_to =\> "multiline\_listener" > } > } multiline.yml > input { > pipeline {address =\> "multiline\_listener"} > codec =\> multiline {blah blah ..} > } > filter {will parse here} > output {will send output to the elasticsearch etc...} Here my ambiguity is about multiline.yml file. I'm adding two things in input, 1st is the input listener and 2nd one is the codec. Don't know it will work or not. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 24, 2020, 7:28pm UTC](https://discuss.elastic.co/t/multiline-codec-in-filter/256520/8 "2020-11-24T19:28:01Z")
pipeline to pipeline links ignore the codec, since they communicate using the pipeline bus, that is why I said to use tcp. ---
**Author:** ![shani](https://avatars.discourse-cdn.com/v4/letter/s/e95f7d/32.png) [@shani](https://discuss.elastic.co/u/shani)\ **Post date:** [November 24, 2020, 7:29pm UTC](https://discuss.elastic.co/t/multiline-codec-in-filter/256520/9 "2020-11-24T19:29:41Z")
If you could please add an example I would be greatful. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 24, 2020, 7:47pm UTC](https://discuss.elastic.co/t/multiline-codec-in-filter/256520/10 "2020-11-24T19:47:41Z")
There is an example [here](https://discuss.elastic.co/t/filter-cef/181215/5). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 22, 2020, 7:47pm UTC](https://discuss.elastic.co/t/multiline-codec-in-filter/256520/11 "2020-12-22T19:47:52Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.