# Multiline codec behavior on large input files

**URL:** <https://discuss.elastic.co/t/multiline-codec-behavior-on-large-input-files/355838>\
**Category:** Logstash\
**Created:** [March 20, 2024, 5:43pm UTC](https://discuss.elastic.co/t/multiline-codec-behavior-on-large-input-files/355838 "2024-03-20T17:43:14Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mahdi\_Moazami](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mahdi_moazami/32/113304_2.png) [@Mahdi\_Moazami](https://discuss.elastic.co/u/Mahdi_Moazami)\
**Post date:** [March 20, 2024, 5:43pm UTC](https://discuss.elastic.co/t/multiline-codec-behavior-on-large-input-files/355838/1 "2024-03-20T17:43:14Z")

</div>

Hi there  
hope you're doing well.  
I have a question about the details about how multiline codec processes the input file. based on the file input docs, the file is chunked and lines are being read from the chunks. say we have a large input file (about 5Gb size) in read mode, and there is a pattern in multiline codec to merge lines into one event. my question is that if the codec reads through a chunk and still **has not reached** the [max\_bytes](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html#plugins-codecs-multiline-max_bytes) or [max\_lines](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html#plugins-codecs-multiline-max_lines) thresholds, but in the end of current chunk the pattern is not still met, what is the behavior of the multiline codec? does it continue to the next chunk looking for the pattern or it terminates the merged lines and start over the next chunk?  
Actually I want to know that is it necessary to alter the default [chunk\_size](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-file_chunk_size) in file input if the input file is very large in size and also the pattern for multiline codec may need to contains 1000 or more lines to be merged into single event?

My input is the following and I'm not sure if I really need to consider a big number for **file\_chunk\_size** option based on the descriptions above.

```auto
input {
    file {
        mode => "read"
        path => [...]
        file_chunk_size => 1024000
        codec => multiline {
            pattern => 'WARC-Type: request'
            negate => true
            what => previous
            auto_flush_interval => 10
            max_lines => 100000
            charset => "UTF-8"
        }
        file_completed_action => "log_and_delete"
        file_completed_log_path => ...
    }
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 20, 2024, 9:04pm UTC](https://discuss.elastic.co/t/multiline-codec-behavior-on-large-input-files/355838/2 "2024-03-20T21:04:18Z")

</div>

> [@Mahdi\_Moazami](#):
>
> my question is that if the codec reads through a chunk and still **has not reached** the [max\_bytes](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html#plugins-codecs-multiline-max_bytes) or [max\_lines](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html#plugins-codecs-multiline-max_lines) thresholds, but in the end of current chunk the pattern is not still met, what is the behavior of the multiline codec?

chunk processing is way upstream of the codec. The file input uses the filewatch library to read the file. filewatch reads a chunk and splits it into lines. If a chunk doesn't end with a line delimiter it continues reading the line from the next chunk. Once it has a complete line it passes it to the file input. The file input then passes the line to the multiline codec.

I cannot see any way for the chunk size to affect the multiline codec.

---

<div class="post-metadata">

**Author:** ![Mahdi\_Moazami](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mahdi_moazami/32/113304_2.png) [@Mahdi\_Moazami](https://discuss.elastic.co/u/Mahdi_Moazami)\
**Post date:** [March 20, 2024, 9:31pm UTC](https://discuss.elastic.co/t/multiline-codec-behavior-on-large-input-files/355838/3 "2024-03-20T21:31:00Z")

</div>

Thanks Badger. Now I see the workflow and the independency of codec with file read process 👍

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 17, 2024, 9:31pm UTC](https://discuss.elastic.co/t/multiline-codec-behavior-on-large-input-files/355838/4 "2024-04-17T21:31:37Z")

</div>

Closed on April 17, 2024, 9:31pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.