# Logstash using multiline for big log file

**URL:** <https://discuss.elastic.co/t/logstash-using-multiline-for-big-log-file/292150>\
**Category:** Logstash\
**Created:** [December 16, 2021, 12:29pm UTC](https://discuss.elastic.co/t/logstash-using-multiline-for-big-log-file/292150 "2021-12-16T12:29:25Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![pan1](https://avatars.discourse-cdn.com/v4/letter/p/e495f1/32.png) [@pan1](https://discuss.elastic.co/u/pan1)\
**Post date:** [December 16, 2021, 12:29pm UTC](https://discuss.elastic.co/t/logstash-using-multiline-for-big-log-file/292150/1 "2021-12-16T12:29:25Z")

</div>

Hello everyone,

I have to handle big log-files (arround 50k lines) using ELK and want to extract some information out of it.  
A long story short, I want to use filter for searching for specific informations, store those infos and drop the rest. Therefore i used a logstash configuration like this:

```auto
input {
  file{
    type => "test" 
    path => "/usr/share/logstash/input/*/log"
    start_position => "beginning"
    codec => multiline {
            pattern => "Finished: (SUCCESS|FAILURE)"
            negate => "true"
            what => "next"
            max_lines => 16000
            max_bytes => "30MiB"
    }
  }
}

filter {
    grok {
        patterns_dir => ["/usr/share/logstash/patterns"]
        match => { "message" => "Finished: %{WORD:build_state}" }
        match => { "message" => ".*checkout in %{NODE_NAME:nodes}.*" }
    }
    mutate {
        remove_field => ["message"]
    }

}
output {
   elasticsearch {
        hosts => "elasticsearch"
    }
}

```

The idea is: every file ends with either "Finished: SUCCESS" or "Finished: FAILURE", so I add every line before to a multiline event. Afterwards I match some informations like, id, cluster node... It all works fine for small files, so the configuration above works perfectly for me and files smaller than 16k lines, however when I increase the max\_lines to 32k for example it does not match the pattern "._checkout in %{NODE\_NAME:nodes}._" anymore.  
What happens there how can I fix that issue? Is it possible to filter the input like "just read lines matching ..." to decrease data amount?

Kind regards

Philip

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [December 16, 2021, 4:08pm UTC](https://discuss.elastic.co/t/logstash-using-multiline-for-big-log-file/292150/2 "2021-12-16T16:08:31Z")

</div>

> [@pan1](#):
>
> Is it possible to filter the input like "just read lines matching ..." to decrease data amount?

Not in logstash. filebeat can do that with the [include\_lines](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#filebeat-input-filestream-include-lines) option.

---

<div class="post-metadata">

**Author:** ![pan1](https://avatars.discourse-cdn.com/v4/letter/p/e495f1/32.png) [@pan1](https://discuss.elastic.co/u/pan1)\
**Post date:** [December 17, 2021, 7:48am UTC](https://discuss.elastic.co/t/logstash-using-multiline-for-big-log-file/292150/3 "2021-12-17T07:48:33Z")

</div>

ok thanks, but do you know why my setup does not work for bigger files ?  
Another idea was to combine the events afterwards, is there a way to merge multiple logs to a multiline log using Elasticsearch?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [December 17, 2021, 5:18pm UTC](https://discuss.elastic.co/t/logstash-using-multiline-for-big-log-file/292150/4 "2021-12-17T17:18:50Z")

</div>

> [@pan1](#):
>
> ok thanks, but do you know why my setup does not work for bigger files ?

No, I do not.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 14, 2022, 5:19pm UTC](https://discuss.elastic.co/t/logstash-using-multiline-for-big-log-file/292150/5 "2022-01-14T17:19:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.