# Logstash pipeline StackOverflowError when using large conf file

**URL:** <https://discuss.elastic.co/t/logstash-pipeline-stackoverflowerror-when-using-large-conf-file/348471>\
**Category:** Logstash\
**Created:** [December 2, 2023, 11:21am UTC](https://discuss.elastic.co/t/logstash-pipeline-stackoverflowerror-when-using-large-conf-file/348471 "2023-12-02T11:21:33Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![blardy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blardy/32/128468_2.png) [@blardy](https://discuss.elastic.co/u/blardy)\
**Post date:** [December 2, 2023, 11:21am UTC](https://discuss.elastic.co/t/logstash-pipeline-stackoverflowerror-when-using-large-conf-file/348471/1 "2023-12-02T11:21:33Z")

</div>

Hey there,

Is there a limitation for logstash regarding the size of the configuration file ? I am having StackOverflowError error when logstash starts when using a large conf file (like 400kb) .

Below an excerpt of the error:

```auto
[INFO] 2023-12-02 11:06:01.398 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>16, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2000, "pipeline.sources"=>["/data/test2.conf"], :thread=>"#<Thread:0x7bc25d02 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[FATAL] 2023-12-02 11:06:01.438 [Ruby-0-Thread-11: /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:289] Logstash - uncaught error (in thread Ruby-0-Thread-11: /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:289)
java.lang.StackOverflowError: null
        at java.util.stream.ReduceOps$3ReducingSink.begin(java/util/stream/ReduceOps.java:164) ~[?:?]
        at java.util.stream.Sink$ChainedReference.begin(java/util/stream/Sink.java:253) ~[?:?]
        at java.util.stream.ReferencePipeline$2$1.begin(java/util/stream/ReferencePipeline.java:173) ~[?:?]
        at java.util.stream.Sink$ChainedReference.begin(java/util/stream/Sink.java:253) ~[?

```

Anyone can indicate how do we handle large conf file ? 😄

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [December 2, 2023, 1:09pm UTC](https://discuss.elastic.co/t/logstash-pipeline-stackoverflowerror-when-using-large-conf-file/348471/2 "2023-12-02T13:09:50Z")

</div>

Welcome to the community.

What did you put inside? Another app?!  
Yes you can, split your logic and push to several [pipelines](https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html) which process your data.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [December 2, 2023, 2:10pm UTC](https://discuss.elastic.co/t/logstash-pipeline-stackoverflowerror-when-using-large-conf-file/348471/3 "2023-12-02T14:10:55Z")

</div>

> [@blardy](#):
>
> Is there a limitation for logstash regarding the size of the configuration file ?

I don't think there is a limitation, it may be something in your pipeline that is not right, but a 400kb text file for just one pipeline seems pretty big.

Can you share your configuration on a github gist?

---

<div class="post-metadata">

**Author:** ![blardy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blardy/32/128468_2.png) [@blardy](https://discuss.elastic.co/u/blardy)\
**Post date:** [December 2, 2023, 3:27pm UTC](https://discuss.elastic.co/t/logstash-pipeline-stackoverflowerror-when-using-large-conf-file/348471/4 "2023-12-02T15:27:03Z")

</div>

Yeah I know it is big, Im having multiple conf file in a directory that are all being loaded and in the end it crashes.

here is a single dummy file ~169kb which mimic the crash: [Logstash Big config multiple filters · GitHub](https://gist.github.com/blardy/0d79fb4451d71ed4f9663d4005a70fd0)

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [December 2, 2023, 3:53pm UTC](https://discuss.elastic.co/t/logstash-pipeline-stackoverflowerror-when-using-large-conf-file/348471/5 "2023-12-02T15:53:02Z")

</div>

Yeah, it crashed for me as well run I run that pipeline.

The error with `java.lang.StackOverflowError` is associated with exhaustion of resources.

This [github issue](https://github.com/elastic/logstash/issues/10131) has a solution, you need to increase the stack size used by each thread.

For me this pipeline only worked after I set `-Xss2M` in `jvm.options`, you may need to use higher values.

Also, depending on how your pipelines looks like, maybe you can optimize then or even use multiple logstash servers to process your data in parallel.

---

<div class="post-metadata">

**Author:** ![blardy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blardy/32/128468_2.png) [@blardy](https://discuss.elastic.co/u/blardy)\
**Post date:** [December 2, 2023, 5:08pm UTC](https://discuss.elastic.co/t/logstash-pipeline-stackoverflowerror-when-using-large-conf-file/348471/6 "2023-12-02T17:08:10Z")

</div>

Thank you ! That indeed solved it 😍

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 30, 2023, 5:08pm UTC](https://discuss.elastic.co/t/logstash-pipeline-stackoverflowerror-when-using-large-conf-file/348471/7 "2023-12-30T17:08:21Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.